status-im
/
nimbus-eth2
mirror of https://github.com/status-im/nimbus-eth2.git
2
0
Fork 0
Code Issues Projects Releases Wiki Activity
nimbus-eth2/ncli/ncli_common.nim

502 lines
20 KiB
Nim
Raw Permalink Normal View History

proc -> func, mainly in spec/state transition and adjecent modules (#3405)
2022-02-17 11:53:55 +00:00
# beacon_chain
verify `genesis_time` more strictly (fixes #1667) (#5694) Bogus values lead to crashes down the line when timers overflow
2024-01-06 14:26:56 +00:00
# Copyright (c) 2022-2024 Status Research & Development GmbH
proc -> func, mainly in spec/state transition and adjecent modules (#3405)
2022-02-17 11:53:55 +00:00
# Licensed and distributed under either of
# * MIT license (license terms in the root directory or at https://opensource.org/licenses/MIT).
# * Apache v2 license (license terms in the root directory or at https://www.apache.org/licenses/LICENSE-2.0).
# at your option. This file may not be copied, modified, or distributed except according to those terms.
extend `ncli` exception handling (#5905) Make raised exceptions explicit in `ncli_common.nim`, and handle more of them in `ncli_db.nim` to have better UX when directories cannot be read or file names do not parse against the expected format.
2024-02-19 09:56:19 +00:00
{.push raises: [].}
More efficient reward data persistance; Address review comments The new format is based on compressed CSV files in two channels: * Detailed per-epoch data * Aggregated "daily" summaries The use of append-only CSV file speeds up significantly the epoch processing speed during data generation. The use of compression results in smaller storage requirements overall. The use of the aggregated files has a very minor cost in both CPU and storage, but leads to near interactive speed for report generation. Other changes: - Implemented support for graceful shut downs to avoid corrupting the saved files. - Fixed a memory leak caused by lacking `StateCache` clean up on each iteration. - Addressed review comments - Moved the rewards and penalties calculation code in a separate module Required invasive changes to existing modules: - The `data` field of the `KeyedBlockRef` type is made public to be used by the validator rewards monitor's Chain DAG update procedure. - The `getForkedBlock` procedure from the `blockchain_dag.nim` module is made public to be used by the validator rewards monitor's Chain DAG update procedure.
2022-01-17 12:58:33 +00:00
import
remove ncli `re` import/`libpcre` dependency (#4208)
2022-10-03 21:43:40 +00:00
std/[os, strutils],
Bugfix: Take into account the finalization delay in the ncli_db rewards calculation This fixes a reward calculation error affecting Prater's epoch 31256
2022-01-23 19:41:29 +00:00
stew/bitops2,
More efficient reward data persistance; Address review comments The new format is based on compressed CSV files in two channels: * Detailed per-epoch data * Aggregated "daily" summaries The use of append-only CSV file speeds up significantly the epoch processing speed during data generation. The use of compression results in smaller storage requirements overall. The use of the aggregated files has a very minor cost in both CPU and storage, but leads to near interactive speed for report generation. Other changes: - Implemented support for graceful shut downs to avoid corrupting the saved files. - Fixed a memory leak caused by lacking `StateCache` clean up on each iteration. - Addressed review comments - Moved the rewards and penalties calculation code in a separate module Required invasive changes to existing modules: - The `data` field of the `KeyedBlockRef` type is made public to be used by the validator rewards monitor's Chain DAG update procedure. - The `getForkedBlock` procedure from the `blockchain_dag.nim` module is made public to be used by the validator rewards monitor's Chain DAG update procedure.
2022-01-17 12:58:33 +00:00
../beacon_chain/spec/[
datatypes/base,
datatypes/phase0,
datatypes/altair,
complete switch to beacon_chain/specs/datatypes/bellatrix (#3295)
2022-01-18 13:36:52 +00:00
datatypes/bellatrix,
More efficient reward data persistance; Address review comments The new format is based on compressed CSV files in two channels: * Detailed per-epoch data * Aggregated "daily" summaries The use of append-only CSV file speeds up significantly the epoch processing speed during data generation. The use of compression results in smaller storage requirements overall. The use of the aggregated files has a very minor cost in both CPU and storage, but leads to near interactive speed for report generation. Other changes: - Implemented support for graceful shut downs to avoid corrupting the saved files. - Fixed a memory leak caused by lacking `StateCache` clean up on each iteration. - Addressed review comments - Moved the rewards and penalties calculation code in a separate module Required invasive changes to existing modules: - The `data` field of the `KeyedBlockRef` type is made public to be used by the validator rewards monitor's Chain DAG update procedure. - The `getForkedBlock` procedure from the `blockchain_dag.nim` module is made public to be used by the validator rewards monitor's Chain DAG update procedure.
2022-01-17 12:58:33 +00:00
beaconstate,
state_transition_epoch,
state_transition_block,
signatures],
Prune `BlockRef` on finalization (#3513) Up til now, the block dag has been using `BlockRef`, a structure adapted for a full DAG, to represent all of chain history. This is a correct and simple design, but does not exploit the linearity of the chain once parts of it finalize. By pruning the in-memory `BlockRef` structure at finalization, we save, at the time of writing, a cool ~250mb (or 25%:ish) chunk of memory landing us at a steady state of ~750mb normal memory usage for a validating node. Above all though, we prevent memory usage from growing proportionally with the length of the chain, something that would not be sustainable over time - instead, the steady state memory usage is roughly determined by the validator set size which grows much more slowly. With these changes, the core should remain sustainable memory-wise post-merge all the way to withdrawals (when the validator set is expected to grow). In-memory indices are still used for the "hot" unfinalized portion of the chain - this ensure that consensus performance remains unchanged. What changes is that for historical access, we use a db-based linear slot index which is cache-and-disk-friendly, keeping the cost for accessing historical data at a similar level as before, achieving the savings at no percievable cost to functionality or performance. A nice collateral benefit is the almost-instant startup since we no longer load any large indicies at dag init. The cost of this functionality instead can be found in the complexity of having to deal with two ways of traversing the chain - by `BlockRef` and by slot. * use `BlockId` instead of `BlockRef` where finalized / historical data may be required * simplify clearance pre-advancement * remove dag.finalizedBlocks (~50:ish mb) * remove `getBlockAtSlot` - use `getBlockIdAtSlot` instead * `parent` and `atSlot` for `BlockId` now require a `ChainDAGRef` instance, unlike `BlockRef` traversal * prune `BlockRef` parents on finality (~200:ish mb) * speed up ChainDAG init by not loading finalized history index * mess up light client server error handling - this need revisiting :)
2022-03-17 17:42:56 +00:00
../beacon_chain/consensus_object_pools/blockchain_dag
More efficient reward data persistance; Address review comments The new format is based on compressed CSV files in two channels: * Detailed per-epoch data * Aggregated "daily" summaries The use of append-only CSV file speeds up significantly the epoch processing speed during data generation. The use of compression results in smaller storage requirements overall. The use of the aggregated files has a very minor cost in both CPU and storage, but leads to near interactive speed for report generation. Other changes: - Implemented support for graceful shut downs to avoid corrupting the saved files. - Fixed a memory leak caused by lacking `StateCache` clean up on each iteration. - Addressed review comments - Moved the rewards and penalties calculation code in a separate module Required invasive changes to existing modules: - The `data` field of the `KeyedBlockRef` type is made public to be used by the validator rewards monitor's Chain DAG update procedure. - The `getForkedBlock` procedure from the `blockchain_dag.nim` module is made public to be used by the validator rewards monitor's Chain DAG update procedure.
2022-01-17 12:58:33 +00:00
add Capella support to Forked* (#4276) * add Capella support to Forked* * remove cruft * add `OnForkyBlockAdded`
2022-11-02 16:23:30 +00:00
from ../beacon_chain/spec/datatypes/capella import BeaconState
More efficient reward data persistance; Address review comments The new format is based on compressed CSV files in two channels: * Detailed per-epoch data * Aggregated "daily" summaries The use of append-only CSV file speeds up significantly the epoch processing speed during data generation. The use of compression results in smaller storage requirements overall. The use of the aggregated files has a very minor cost in both CPU and storage, but leads to near interactive speed for report generation. Other changes: - Implemented support for graceful shut downs to avoid corrupting the saved files. - Fixed a memory leak caused by lacking `StateCache` clean up on each iteration. - Addressed review comments - Moved the rewards and penalties calculation code in a separate module Required invasive changes to existing modules: - The `data` field of the `KeyedBlockRef` type is made public to be used by the validator rewards monitor's Chain DAG update procedure. - The `getForkedBlock` procedure from the `blockchain_dag.nim` module is made public to be used by the validator rewards monitor's Chain DAG update procedure.
2022-01-17 12:58:33 +00:00
type
RewardsAndPenalties* = object
source_outcome*: int64
max_source_reward*: Gwei
target_outcome*: int64
max_target_reward*: Gwei
head_outcome*: int64
max_head_reward*: Gwei
inclusion_delay_outcome*: int64
max_inclusion_delay_reward*: Gwei
sync_committee_outcome*: int64
max_sync_committee_reward*: Gwei
proposer_outcome*: int64
inactivity_penalty*: Gwei
slashing_outcome*: int64
deposits*: Gwei
inclusion_delay*: Option[uint64]
ParticipationFlags* = object
currentEpochParticipation: EpochParticipationFlags
previousEpochParticipation: EpochParticipationFlags
document and clean up `ValidatorIndex` usage (#3651) * document static vs dynamic range checking requirements * add `vindices` iterator to iterate over valid validator indices in a state * clean up spec comments in general * fixup Co-authored-by: tersec <tersec@users.noreply.github.com>
2022-05-23 23:39:08 +00:00
PubkeyToIndexTable = Table[ValidatorPubKey, ValidatorIndex]
More efficient reward data persistance; Address review comments The new format is based on compressed CSV files in two channels: * Detailed per-epoch data * Aggregated "daily" summaries The use of append-only CSV file speeds up significantly the epoch processing speed during data generation. The use of compression results in smaller storage requirements overall. The use of the aggregated files has a very minor cost in both CPU and storage, but leads to near interactive speed for report generation. Other changes: - Implemented support for graceful shut downs to avoid corrupting the saved files. - Fixed a memory leak caused by lacking `StateCache` clean up on each iteration. - Addressed review comments - Moved the rewards and penalties calculation code in a separate module Required invasive changes to existing modules: - The `data` field of the `KeyedBlockRef` type is made public to be used by the validator rewards monitor's Chain DAG update procedure. - The `getForkedBlock` procedure from the `blockchain_dag.nim` module is made public to be used by the validator rewards monitor's Chain DAG update procedure.
2022-01-17 12:58:33 +00:00
AuxiliaryState* = object
epochParticipationFlags: ParticipationFlags
pubkeyToIndex: PubkeyToIndexTable
const
epochInfoFileNameDigitsCount = 8
remove ncli `re` import/`libpcre` dependency (#4208)
2022-10-03 21:43:40 +00:00
epochFileNameExtension = ".epoch"
More efficient reward data persistance; Address review comments The new format is based on compressed CSV files in two channels: * Detailed per-epoch data * Aggregated "daily" summaries The use of append-only CSV file speeds up significantly the epoch processing speed during data generation. The use of compression results in smaller storage requirements overall. The use of the aggregated files has a very minor cost in both CPU and storage, but leads to near interactive speed for report generation. Other changes: - Implemented support for graceful shut downs to avoid corrupting the saved files. - Fixed a memory leak caused by lacking `StateCache` clean up on each iteration. - Addressed review comments - Moved the rewards and penalties calculation code in a separate module Required invasive changes to existing modules: - The `data` field of the `KeyedBlockRef` type is made public to be used by the validator rewards monitor's Chain DAG update procedure. - The `getForkedBlock` procedure from the `blockchain_dag.nim` module is made public to be used by the validator rewards monitor's Chain DAG update procedure.
2022-01-17 12:58:33 +00:00
proc -> func, mainly in spec/state transition and adjecent modules (#3405)
2022-02-17 11:53:55 +00:00
func copyParticipationFlags*(auxiliaryState: var AuxiliaryState,
More efficient reward data persistance; Address review comments The new format is based on compressed CSV files in two channels: * Detailed per-epoch data * Aggregated "daily" summaries The use of append-only CSV file speeds up significantly the epoch processing speed during data generation. The use of compression results in smaller storage requirements overall. The use of the aggregated files has a very minor cost in both CPU and storage, but leads to near interactive speed for report generation. Other changes: - Implemented support for graceful shut downs to avoid corrupting the saved files. - Fixed a memory leak caused by lacking `StateCache` clean up on each iteration. - Addressed review comments - Moved the rewards and penalties calculation code in a separate module Required invasive changes to existing modules: - The `data` field of the `KeyedBlockRef` type is made public to be used by the validator rewards monitor's Chain DAG update procedure. - The `getForkedBlock` procedure from the `blockchain_dag.nim` module is made public to be used by the validator rewards monitor's Chain DAG update procedure.
2022-01-17 12:58:33 +00:00
forkedState: ForkedHashedBeaconState) =
withState(forkedState):
rename `stateFork` > `consensusFork` (#4718) Just the variable, not yet `lcDataForkAtStateFork` / `atStateFork`. - Shorten comment in `light_client.nim` to keep line width - Do not rename `stateFork` mention in `runProposalForkchoiceUpdated`. - Do not rename `stateFork` in `getStateField(dag.headState, fork)` Rest is just a mechanical mass replace
2023-03-11 00:35:52 +00:00
when consensusFork > ConsensusFork.Phase0:
More efficient reward data persistance; Address review comments The new format is based on compressed CSV files in two channels: * Detailed per-epoch data * Aggregated "daily" summaries The use of append-only CSV file speeds up significantly the epoch processing speed during data generation. The use of compression results in smaller storage requirements overall. The use of the aggregated files has a very minor cost in both CPU and storage, but leads to near interactive speed for report generation. Other changes: - Implemented support for graceful shut downs to avoid corrupting the saved files. - Fixed a memory leak caused by lacking `StateCache` clean up on each iteration. - Addressed review comments - Moved the rewards and penalties calculation code in a separate module Required invasive changes to existing modules: - The `data` field of the `KeyedBlockRef` type is made public to be used by the validator rewards monitor's Chain DAG update procedure. - The `getForkedBlock` procedure from the `blockchain_dag.nim` module is made public to be used by the validator rewards monitor's Chain DAG update procedure.
2022-01-17 12:58:33 +00:00
template flags: untyped = auxiliaryState.epochParticipationFlags
more `withState` `state` -> `forkyState` (#4112)
2022-09-13 11:53:12 +00:00
flags.currentEpochParticipation =
forkyState.data.current_epoch_participation
flags.previousEpochParticipation =
forkyState.data.previous_epoch_participation
More efficient reward data persistance; Address review comments The new format is based on compressed CSV files in two channels: * Detailed per-epoch data * Aggregated "daily" summaries The use of append-only CSV file speeds up significantly the epoch processing speed during data generation. The use of compression results in smaller storage requirements overall. The use of the aggregated files has a very minor cost in both CPU and storage, but leads to near interactive speed for report generation. Other changes: - Implemented support for graceful shut downs to avoid corrupting the saved files. - Fixed a memory leak caused by lacking `StateCache` clean up on each iteration. - Addressed review comments - Moved the rewards and penalties calculation code in a separate module Required invasive changes to existing modules: - The `data` field of the `KeyedBlockRef` type is made public to be used by the validator rewards monitor's Chain DAG update procedure. - The `getForkedBlock` procedure from the `blockchain_dag.nim` module is made public to be used by the validator rewards monitor's Chain DAG update procedure.
2022-01-17 12:58:33 +00:00
remove ncli `re` import/`libpcre` dependency (#4208)
2022-10-03 21:43:40 +00:00
from std/sequtils import allIt
func matchFilenameUnaggregatedFiles(filename: string): bool =
# epochNumberRegexStr & epochFileNameExtension
filename.len == epochInfoFileNameDigitsCount + epochFileNameExtension.len and
filename.endsWith(epochFileNameExtension) and
# TODO should use toOpenArray, but
# https://github.com/nim-lang/Nim/issues/15952
# https://github.com/nim-lang/Nim/issues/19969
allIt(filename[0 ..< epochInfoFileNameDigitsCount], it.isDigit)
static:
for filename in [
"00000000.epoch", "00243929.epoch", "04957024.epoch", "39820353.epoch",
"82829191.epoch", "85740966.epoch", "93321944.epoch", "98928899.epoch"]:
doAssert filename.matchFilenameUnaggregatedFiles
for filename in [
# Valid aggregated, not unaggregated
"03820350_13372742.epoch", "04117778_69588614.epoch",
"25249017_64218993.epoch", "34265267_41589365.epoch",
"57926659_59282297.epoch", "67699314_92835461.epoch",
"0000000.epoch", # Too short
"000000000.epoch", # Too long
"75787x73.epoch", # Incorrect number format
"00000000.ecpoh"]: # Wrong extension
doAssert not filename.matchFilenameUnaggregatedFiles
func matchFilenameAggregatedFiles(filename: string): bool =
# epochNumberRegexStr & "_" & epochNumberRegexStr & epochFileNameExtension
filename.len == epochInfoFileNameDigitsCount * 2 + "_".len + epochFileNameExtension.len and
filename.endsWith(epochFileNameExtension) and
# TODO should use toOpenArray, but
# https://github.com/nim-lang/Nim/issues/15952
# https://github.com/nim-lang/Nim/issues/19969
allIt(filename[0 ..< epochInfoFileNameDigitsCount], it.isDigit) and
filename[epochInfoFileNameDigitsCount] == '_' and
allIt(
filename[epochInfoFileNameDigitsCount + 1 ..< 2 * epochInfoFileNameDigitsCount + 1],
it.isDigit)
static:
for filename in [
"03820350_13372742.epoch", "04117778_69588614.epoch",
"25249017_64218993.epoch", "34265267_41589365.epoch",
"57926659_59282297.epoch", "67699314_92835461.epoch"]:
doAssert filename.matchFilenameAggregatedFiles
for filename in [
# Valid unaggregated, not aggregated
"00000000.epoch", "00243929.epoch", "04957024.epoch", "39820353.epoch",
"82829191.epoch", "85740966.epoch", "93321944.epoch", "98928899.epoch",
"00000000_0000000.epoch", # Too short
"31x85971_93149672.epoch", # Incorrect number format, first field
"18049105&72034596.epoch", # No underscore separator
"31485971_931496x2.epoch", # Incorrect number format, second field
"15227487_86601706.echop"]: # Wrong extension
doAssert not filename.matchFilenameAggregatedFiles
fix most ConvFromXtoItselfNotNeeded hints and unhide remaining ones (#6307)
2024-05-22 11:56:37 +00:00
proc getUnaggregatedFilesEpochRange*(dir: string):
tuple[firstEpoch, lastEpoch: Epoch] {.raises: [OSError, ValueError].} =
More efficient reward data persistance; Address review comments The new format is based on compressed CSV files in two channels: * Detailed per-epoch data * Aggregated "daily" summaries The use of append-only CSV file speeds up significantly the epoch processing speed during data generation. The use of compression results in smaller storage requirements overall. The use of the aggregated files has a very minor cost in both CPU and storage, but leads to near interactive speed for report generation. Other changes: - Implemented support for graceful shut downs to avoid corrupting the saved files. - Fixed a memory leak caused by lacking `StateCache` clean up on each iteration. - Addressed review comments - Moved the rewards and penalties calculation code in a separate module Required invasive changes to existing modules: - The `data` field of the `KeyedBlockRef` type is made public to be used by the validator rewards monitor's Chain DAG update procedure. - The `getForkedBlock` procedure from the `blockchain_dag.nim` module is made public to be used by the validator rewards monitor's Chain DAG update procedure.
2022-01-17 12:58:33 +00:00
var smallestEpochFileName =
'9'.repeat(epochInfoFileNameDigitsCount) & epochFileNameExtension
var largestEpochFileName =
'0'.repeat(epochInfoFileNameDigitsCount) & epochFileNameExtension
fix most ConvFromXtoItselfNotNeeded hints and unhide remaining ones (#6307)
2024-05-22 11:56:37 +00:00
for (_, fn) in walkDir(dir, relative = true):
remove ncli `re` import/`libpcre` dependency (#4208)
2022-10-03 21:43:40 +00:00
if fn.matchFilenameUnaggregatedFiles:
More efficient reward data persistance; Address review comments The new format is based on compressed CSV files in two channels: * Detailed per-epoch data * Aggregated "daily" summaries The use of append-only CSV file speeds up significantly the epoch processing speed during data generation. The use of compression results in smaller storage requirements overall. The use of the aggregated files has a very minor cost in both CPU and storage, but leads to near interactive speed for report generation. Other changes: - Implemented support for graceful shut downs to avoid corrupting the saved files. - Fixed a memory leak caused by lacking `StateCache` clean up on each iteration. - Addressed review comments - Moved the rewards and penalties calculation code in a separate module Required invasive changes to existing modules: - The `data` field of the `KeyedBlockRef` type is made public to be used by the validator rewards monitor's Chain DAG update procedure. - The `getForkedBlock` procedure from the `blockchain_dag.nim` module is made public to be used by the validator rewards monitor's Chain DAG update procedure.
2022-01-17 12:58:33 +00:00
if fn < smallestEpochFileName:
smallestEpochFileName = fn
if fn > largestEpochFileName:
largestEpochFileName = fn
result.firstEpoch = parseUInt(
smallestEpochFileName[0 ..< epochInfoFileNameDigitsCount]).Epoch
result.lastEpoch = parseUInt(
largestEpochFileName[0 ..< epochInfoFileNameDigitsCount]).Epoch
extend `ncli` exception handling (#5905) Make raised exceptions explicit in `ncli_common.nim`, and handle more of them in `ncli_db.nim` to have better UX when directories cannot be read or file names do not parse against the expected format.
2024-02-19 09:56:19 +00:00
proc getUnaggregatedFilesLastEpoch*(
dir: string): Epoch {.raises: [OSError, ValueError].} =
Streamline the validator reward analysis Notable improvements: * A separate aggregation pass is no longer required. * The user can opt to produce only aggregated data (resuing in a much smaller data set). * Large portion of the number cruching in Jupyter is now done in C through the rich DataFrames API. * Added support for comparisons against the "median" validator performance in the network.
2022-01-31 12:06:16 +00:00
dir.getUnaggregatedFilesEpochRange.lastEpoch
extend `ncli` exception handling (#5905) Make raised exceptions explicit in `ncli_common.nim`, and handle more of them in `ncli_db.nim` to have better UX when directories cannot be read or file names do not parse against the expected format.
2024-02-19 09:56:19 +00:00
proc getAggregatedFilesLastEpoch*(
dir: string): Epoch {.raises: [OSError, ValueError].}=
Streamline the validator reward analysis Notable improvements: * A separate aggregation pass is no longer required. * The user can opt to produce only aggregated data (resuing in a much smaller data set). * Large portion of the number cruching in Jupyter is now done in C through the rich DataFrames API. * Added support for comparisons against the "median" validator performance in the network.
2022-01-31 12:06:16 +00:00
var largestEpochInFileName = 0'u
fix most ConvFromXtoItselfNotNeeded hints and unhide remaining ones (#6307)
2024-05-22 11:56:37 +00:00
for (_, fn) in walkDir(dir, relative = true):
remove ncli `re` import/`libpcre` dependency (#4208)
2022-10-03 21:43:40 +00:00
if fn.matchFilenameAggregatedFiles:
stylecheck fixes (#3592)
2022-04-14 10:47:14 +00:00
let fileLastEpoch = parseUInt(
Streamline the validator reward analysis Notable improvements: * A separate aggregation pass is no longer required. * The user can opt to produce only aggregated data (resuing in a much smaller data set). * Large portion of the number cruching in Jupyter is now done in C through the rich DataFrames API. * Added support for comparisons against the "median" validator performance in the network.
2022-01-31 12:06:16 +00:00
fn[epochInfoFileNameDigitsCount + 1 .. 2 * epochInfoFileNameDigitsCount])
if fileLastEpoch > largestEpochInFileName:
largestEpochInFileName = fileLastEpoch
generic putCorruptState; {Merge=>Bellatrix}BeaconStateNoImmutableValidators (#3427)
2022-02-21 11:55:56 +00:00
largestEpochInFileName.Epoch
Streamline the validator reward analysis Notable improvements: * A separate aggregation pass is no longer required. * The user can opt to produce only aggregated data (resuing in a much smaller data set). * Large portion of the number cruching in Jupyter is now done in C through the rich DataFrames API. * Added support for comparisons against the "median" validator performance in the network.
2022-01-31 12:06:16 +00:00
proc -> func, mainly in spec/state transition and adjecent modules (#3405)
2022-02-17 11:53:55 +00:00
func epochAsString*(epoch: Epoch): string =
More efficient reward data persistance; Address review comments The new format is based on compressed CSV files in two channels: * Detailed per-epoch data * Aggregated "daily" summaries The use of append-only CSV file speeds up significantly the epoch processing speed during data generation. The use of compression results in smaller storage requirements overall. The use of the aggregated files has a very minor cost in both CPU and storage, but leads to near interactive speed for report generation. Other changes: - Implemented support for graceful shut downs to avoid corrupting the saved files. - Fixed a memory leak caused by lacking `StateCache` clean up on each iteration. - Addressed review comments - Moved the rewards and penalties calculation code in a separate module Required invasive changes to existing modules: - The `data` field of the `KeyedBlockRef` type is made public to be used by the validator rewards monitor's Chain DAG update procedure. - The `getForkedBlock` procedure from the `blockchain_dag.nim` module is made public to be used by the validator rewards monitor's Chain DAG update procedure.
2022-01-17 12:58:33 +00:00
let strEpoch = $epoch
'0'.repeat(epochInfoFileNameDigitsCount - strEpoch.len) & strEpoch
proc -> func, mainly in spec/state transition and adjecent modules (#3405)
2022-02-17 11:53:55 +00:00
func getFilePathForEpoch*(epoch: Epoch, dir: string): string =
More efficient reward data persistance; Address review comments The new format is based on compressed CSV files in two channels: * Detailed per-epoch data * Aggregated "daily" summaries The use of append-only CSV file speeds up significantly the epoch processing speed during data generation. The use of compression results in smaller storage requirements overall. The use of the aggregated files has a very minor cost in both CPU and storage, but leads to near interactive speed for report generation. Other changes: - Implemented support for graceful shut downs to avoid corrupting the saved files. - Fixed a memory leak caused by lacking `StateCache` clean up on each iteration. - Addressed review comments - Moved the rewards and penalties calculation code in a separate module Required invasive changes to existing modules: - The `data` field of the `KeyedBlockRef` type is made public to be used by the validator rewards monitor's Chain DAG update procedure. - The `getForkedBlock` procedure from the `blockchain_dag.nim` module is made public to be used by the validator rewards monitor's Chain DAG update procedure.
2022-01-17 12:58:33 +00:00
dir / epochAsString(epoch) & epochFileNameExtension
proc -> func, mainly in spec/state transition and adjecent modules (#3405)
2022-02-17 11:53:55 +00:00
func getFilePathForEpochs*(startEpoch, endEpoch: Epoch, dir: string): string =
Streamline the validator reward analysis Notable improvements: * A separate aggregation pass is no longer required. * The user can opt to produce only aggregated data (resuing in a much smaller data set). * Large portion of the number cruching in Jupyter is now done in C through the rich DataFrames API. * Added support for comparisons against the "median" validator performance in the network.
2022-01-31 12:06:16 +00:00
let fileName = epochAsString(startEpoch) & "_" &
epochAsString(endEpoch) & epochFileNameExtension
dir / fileName
Prune `BlockRef` on finalization (#3513) Up til now, the block dag has been using `BlockRef`, a structure adapted for a full DAG, to represent all of chain history. This is a correct and simple design, but does not exploit the linearity of the chain once parts of it finalize. By pruning the in-memory `BlockRef` structure at finalization, we save, at the time of writing, a cool ~250mb (or 25%:ish) chunk of memory landing us at a steady state of ~750mb normal memory usage for a validating node. Above all though, we prevent memory usage from growing proportionally with the length of the chain, something that would not be sustainable over time - instead, the steady state memory usage is roughly determined by the validator set size which grows much more slowly. With these changes, the core should remain sustainable memory-wise post-merge all the way to withdrawals (when the validator set is expected to grow). In-memory indices are still used for the "hot" unfinalized portion of the chain - this ensure that consensus performance remains unchanged. What changes is that for historical access, we use a db-based linear slot index which is cache-and-disk-friendly, keeping the cost for accessing historical data at a similar level as before, achieving the savings at no percievable cost to functionality or performance. A nice collateral benefit is the almost-instant startup since we no longer load any large indicies at dag init. The cost of this functionality instead can be found in the complexity of having to deal with two ways of traversing the chain - by `BlockRef` and by slot. * use `BlockId` instead of `BlockRef` where finalized / historical data may be required * simplify clearance pre-advancement * remove dag.finalizedBlocks (~50:ish mb) * remove `getBlockAtSlot` - use `getBlockIdAtSlot` instead * `parent` and `atSlot` for `BlockId` now require a `ChainDAGRef` instance, unlike `BlockRef` traversal * prune `BlockRef` parents on finality (~200:ish mb) * speed up ChainDAG init by not loading finalized history index * mess up light client server error handling - this need revisiting :)
2022-03-17 17:42:56 +00:00
func getBlockRange*(dag: ChainDAGRef, start, ends: Slot): seq[BlockId] =
More efficient reward data persistance; Address review comments The new format is based on compressed CSV files in two channels: * Detailed per-epoch data * Aggregated "daily" summaries The use of append-only CSV file speeds up significantly the epoch processing speed during data generation. The use of compression results in smaller storage requirements overall. The use of the aggregated files has a very minor cost in both CPU and storage, but leads to near interactive speed for report generation. Other changes: - Implemented support for graceful shut downs to avoid corrupting the saved files. - Fixed a memory leak caused by lacking `StateCache` clean up on each iteration. - Addressed review comments - Moved the rewards and penalties calculation code in a separate module Required invasive changes to existing modules: - The `data` field of the `KeyedBlockRef` type is made public to be used by the validator rewards monitor's Chain DAG update procedure. - The `getForkedBlock` procedure from the `blockchain_dag.nim` module is made public to be used by the validator rewards monitor's Chain DAG update procedure.
2022-01-17 12:58:33 +00:00
# Range of block in reverse order
doAssert start < ends
Prune `BlockRef` on finalization (#3513) Up til now, the block dag has been using `BlockRef`, a structure adapted for a full DAG, to represent all of chain history. This is a correct and simple design, but does not exploit the linearity of the chain once parts of it finalize. By pruning the in-memory `BlockRef` structure at finalization, we save, at the time of writing, a cool ~250mb (or 25%:ish) chunk of memory landing us at a steady state of ~750mb normal memory usage for a validating node. Above all though, we prevent memory usage from growing proportionally with the length of the chain, something that would not be sustainable over time - instead, the steady state memory usage is roughly determined by the validator set size which grows much more slowly. With these changes, the core should remain sustainable memory-wise post-merge all the way to withdrawals (when the validator set is expected to grow). In-memory indices are still used for the "hot" unfinalized portion of the chain - this ensure that consensus performance remains unchanged. What changes is that for historical access, we use a db-based linear slot index which is cache-and-disk-friendly, keeping the cost for accessing historical data at a similar level as before, achieving the savings at no percievable cost to functionality or performance. A nice collateral benefit is the almost-instant startup since we no longer load any large indicies at dag init. The cost of this functionality instead can be found in the complexity of having to deal with two ways of traversing the chain - by `BlockRef` and by slot. * use `BlockId` instead of `BlockRef` where finalized / historical data may be required * simplify clearance pre-advancement * remove dag.finalizedBlocks (~50:ish mb) * remove `getBlockAtSlot` - use `getBlockIdAtSlot` instead * `parent` and `atSlot` for `BlockId` now require a `ChainDAGRef` instance, unlike `BlockRef` traversal * prune `BlockRef` parents on finality (~200:ish mb) * speed up ChainDAG init by not loading finalized history index * mess up light client server error handling - this need revisiting :)
2022-03-17 17:42:56 +00:00
result = newSeqOfCap[BlockId](ends - start)
var current = ends
while current > start:
current -= 1
let bsid = dag.getBlockIdAtSlot(current).valueOr:
continue
if bsid.bid.slot < start: # current might be empty
break
result.add(bsid.bid)
current = bsid.bid.slot # skip empty slots
More efficient reward data persistance; Address review comments The new format is based on compressed CSV files in two channels: * Detailed per-epoch data * Aggregated "daily" summaries The use of append-only CSV file speeds up significantly the epoch processing speed during data generation. The use of compression results in smaller storage requirements overall. The use of the aggregated files has a very minor cost in both CPU and storage, but leads to near interactive speed for report generation. Other changes: - Implemented support for graceful shut downs to avoid corrupting the saved files. - Fixed a memory leak caused by lacking `StateCache` clean up on each iteration. - Addressed review comments - Moved the rewards and penalties calculation code in a separate module Required invasive changes to existing modules: - The `data` field of the `KeyedBlockRef` type is made public to be used by the validator rewards monitor's Chain DAG update procedure. - The `getForkedBlock` procedure from the `blockchain_dag.nim` module is made public to be used by the validator rewards monitor's Chain DAG update procedure.
2022-01-17 12:58:33 +00:00
func getOutcome(delta: RewardDelta): int64 =
delta.rewards.int64 - delta.penalties.int64
proc -> func, mainly in spec/state transition and adjecent modules (#3405)
2022-02-17 11:53:55 +00:00
func collectSlashings(
More efficient reward data persistance; Address review comments The new format is based on compressed CSV files in two channels: * Detailed per-epoch data * Aggregated "daily" summaries The use of append-only CSV file speeds up significantly the epoch processing speed during data generation. The use of compression results in smaller storage requirements overall. The use of the aggregated files has a very minor cost in both CPU and storage, but leads to near interactive speed for report generation. Other changes: - Implemented support for graceful shut downs to avoid corrupting the saved files. - Fixed a memory leak caused by lacking `StateCache` clean up on each iteration. - Addressed review comments - Moved the rewards and penalties calculation code in a separate module Required invasive changes to existing modules: - The `data` field of the `KeyedBlockRef` type is made public to be used by the validator rewards monitor's Chain DAG update procedure. - The `getForkedBlock` procedure from the `blockchain_dag.nim` module is made public to be used by the validator rewards monitor's Chain DAG update procedure.
2022-01-17 12:58:33 +00:00
rewardsAndPenalties: var seq[RewardsAndPenalties],
state: ForkyBeaconState, total_balance: Gwei) =
let
epoch = get_current_epoch(state)
adjusted_total_slashing_balance = get_adjusted_total_slashing_balance(
state, total_balance)
for index in 0 ..< state.validators.len:
fix unnecessary HashList/HashArray cache invalidation (#3660) * SSZ `[]` -> `mitem` * `[]` -> `item` immutable access via mutable instance cannot rely on template overloading, and `[]` cannot be a `func` because of special seq handling in compiler.
2022-05-30 13:30:42 +00:00
let validator = unsafeAddr state.validators[index]
More efficient reward data persistance; Address review comments The new format is based on compressed CSV files in two channels: * Detailed per-epoch data * Aggregated "daily" summaries The use of append-only CSV file speeds up significantly the epoch processing speed during data generation. The use of compression results in smaller storage requirements overall. The use of the aggregated files has a very minor cost in both CPU and storage, but leads to near interactive speed for report generation. Other changes: - Implemented support for graceful shut downs to avoid corrupting the saved files. - Fixed a memory leak caused by lacking `StateCache` clean up on each iteration. - Addressed review comments - Moved the rewards and penalties calculation code in a separate module Required invasive changes to existing modules: - The `data` field of the `KeyedBlockRef` type is made public to be used by the validator rewards monitor's Chain DAG update procedure. - The `getForkedBlock` procedure from the `blockchain_dag.nim` module is made public to be used by the validator rewards monitor's Chain DAG update procedure.
2022-01-17 12:58:33 +00:00
if slashing_penalty_applies(validator[], epoch):
rewardsAndPenalties[index].slashing_outcome +=
validator[].get_slashing_penalty(
adjusted_total_slashing_balance, total_balance).int64
Fix a reward calculation bug affecting Prater epoch 64781 (#3428) To calculate the deltas correctly, the `process_inactivity_updates` function must be called before the rewards and penalties processing code in order to update the `inactivity_scores` field in the state. This would have required duplicating more logic from the spec in the ncli modules, so I've decided to pay the price of introducing a run-time copy of the state at each epoch which eliminates the need to duplicate logic (both for this fix and the previous one). Other changes: * Fixes for the read-only mode of the `BeaconChainDb` * Fix an uint64 underflow in the debug output procedure for printing balance deltas * Allow Bellatrix states in the reward computation helpers
2022-02-22 12:14:17 +00:00
proc collectEpochRewardsAndPenalties*(
More efficient reward data persistance; Address review comments The new format is based on compressed CSV files in two channels: * Detailed per-epoch data * Aggregated "daily" summaries The use of append-only CSV file speeds up significantly the epoch processing speed during data generation. The use of compression results in smaller storage requirements overall. The use of the aggregated files has a very minor cost in both CPU and storage, but leads to near interactive speed for report generation. Other changes: - Implemented support for graceful shut downs to avoid corrupting the saved files. - Fixed a memory leak caused by lacking `StateCache` clean up on each iteration. - Addressed review comments - Moved the rewards and penalties calculation code in a separate module Required invasive changes to existing modules: - The `data` field of the `KeyedBlockRef` type is made public to be used by the validator rewards monitor's Chain DAG update procedure. - The `getForkedBlock` procedure from the `blockchain_dag.nim` module is made public to be used by the validator rewards monitor's Chain DAG update procedure.
2022-01-17 12:58:33 +00:00
rewardsAndPenalties: var seq[RewardsAndPenalties],
Fix a reward calculation bug affecting Prater epoch 64781 (#3428) To calculate the deltas correctly, the `process_inactivity_updates` function must be called before the rewards and penalties processing code in order to update the `inactivity_scores` field in the state. This would have required duplicating more logic from the spec in the ncli modules, so I've decided to pay the price of introducing a run-time copy of the state at each epoch which eliminates the need to duplicate logic (both for this fix and the previous one). Other changes: * Fixes for the read-only mode of the `BeaconChainDb` * Fix an uint64 underflow in the debug output procedure for printing balance deltas * Allow Bellatrix states in the reward computation helpers
2022-02-22 12:14:17 +00:00
state: var phase0.BeaconState, cache: var StateCache, cfg: RuntimeConfig,
Bugfix: Take into account the finalization delay in the ncli_db rewards calculation This fixes a reward calculation error affecting Prater's epoch 31256
2022-01-23 19:41:29 +00:00
flags: UpdateFlags) =
More efficient reward data persistance; Address review comments The new format is based on compressed CSV files in two channels: * Detailed per-epoch data * Aggregated "daily" summaries The use of append-only CSV file speeds up significantly the epoch processing speed during data generation. The use of compression results in smaller storage requirements overall. The use of the aggregated files has a very minor cost in both CPU and storage, but leads to near interactive speed for report generation. Other changes: - Implemented support for graceful shut downs to avoid corrupting the saved files. - Fixed a memory leak caused by lacking `StateCache` clean up on each iteration. - Addressed review comments - Moved the rewards and penalties calculation code in a separate module Required invasive changes to existing modules: - The `data` field of the `KeyedBlockRef` type is made public to be used by the validator rewards monitor's Chain DAG update procedure. - The `getForkedBlock` procedure from the `blockchain_dag.nim` module is made public to be used by the validator rewards monitor's Chain DAG update procedure.
2022-01-17 12:58:33 +00:00
if get_current_epoch(state) == GENESIS_EPOCH:
return
var info: phase0.EpochInfo
info.init(state)
info.process_attestations(state, cache)
doAssert info.validators.len == state.validators.len
rewardsAndPenalties.setLen(state.validators.len)
Fix a reward calculation bug affecting Prater epoch 64781 (#3428) To calculate the deltas correctly, the `process_inactivity_updates` function must be called before the rewards and penalties processing code in order to update the `inactivity_scores` field in the state. This would have required duplicating more logic from the spec in the ncli modules, so I've decided to pay the price of introducing a run-time copy of the state at each epoch which eliminates the need to duplicate logic (both for this fix and the previous one). Other changes: * Fixes for the read-only mode of the `BeaconChainDb` * Fix an uint64 underflow in the debug output procedure for printing balance deltas * Allow Bellatrix states in the reward computation helpers
2022-02-22 12:14:17 +00:00
process_justification_and_finalization(state, info.balances, flags)
More efficient reward data persistance; Address review comments The new format is based on compressed CSV files in two channels: * Detailed per-epoch data * Aggregated "daily" summaries The use of append-only CSV file speeds up significantly the epoch processing speed during data generation. The use of compression results in smaller storage requirements overall. The use of the aggregated files has a very minor cost in both CPU and storage, but leads to near interactive speed for report generation. Other changes: - Implemented support for graceful shut downs to avoid corrupting the saved files. - Fixed a memory leak caused by lacking `StateCache` clean up on each iteration. - Addressed review comments - Moved the rewards and penalties calculation code in a separate module Required invasive changes to existing modules: - The `data` field of the `KeyedBlockRef` type is made public to be used by the validator rewards monitor's Chain DAG update procedure. - The `getForkedBlock` procedure from the `blockchain_dag.nim` module is made public to be used by the validator rewards monitor's Chain DAG update procedure.
2022-01-17 12:58:33 +00:00
let
Fix a reward calculation bug affecting Prater epoch 64781 (#3428) To calculate the deltas correctly, the `process_inactivity_updates` function must be called before the rewards and penalties processing code in order to update the `inactivity_scores` field in the state. This would have required duplicating more logic from the spec in the ncli modules, so I've decided to pay the price of introducing a run-time copy of the state at each epoch which eliminates the need to duplicate logic (both for this fix and the previous one). Other changes: * Fixes for the read-only mode of the `BeaconChainDb` * Fix an uint64 underflow in the debug output procedure for printing balance deltas * Allow Bellatrix states in the reward computation helpers
2022-02-22 12:14:17 +00:00
finality_delay = get_finality_delay(state)
More efficient reward data persistance; Address review comments The new format is based on compressed CSV files in two channels: * Detailed per-epoch data * Aggregated "daily" summaries The use of append-only CSV file speeds up significantly the epoch processing speed during data generation. The use of compression results in smaller storage requirements overall. The use of the aggregated files has a very minor cost in both CPU and storage, but leads to near interactive speed for report generation. Other changes: - Implemented support for graceful shut downs to avoid corrupting the saved files. - Fixed a memory leak caused by lacking `StateCache` clean up on each iteration. - Addressed review comments - Moved the rewards and penalties calculation code in a separate module Required invasive changes to existing modules: - The `data` field of the `KeyedBlockRef` type is made public to be used by the validator rewards monitor's Chain DAG update procedure. - The `getForkedBlock` procedure from the `blockchain_dag.nim` module is made public to be used by the validator rewards monitor's Chain DAG update procedure.
2022-01-17 12:58:33 +00:00
total_balance = info.balances.current_epoch
make `Gwei` `distinct` (#6090) #6087 introduced a subtle change to `nim-web3` resulting in `Gwei` to be serialized differently than before. Using a `distinct` type for `Gwei` improves type safety and avoids such problems in the future.
2024-03-19 13:22:07 +00:00
total_balance_sqrt = integer_squareroot(distinctBase(total_balance))
More efficient reward data persistance; Address review comments The new format is based on compressed CSV files in two channels: * Detailed per-epoch data * Aggregated "daily" summaries The use of append-only CSV file speeds up significantly the epoch processing speed during data generation. The use of compression results in smaller storage requirements overall. The use of the aggregated files has a very minor cost in both CPU and storage, but leads to near interactive speed for report generation. Other changes: - Implemented support for graceful shut downs to avoid corrupting the saved files. - Fixed a memory leak caused by lacking `StateCache` clean up on each iteration. - Addressed review comments - Moved the rewards and penalties calculation code in a separate module Required invasive changes to existing modules: - The `data` field of the `KeyedBlockRef` type is made public to be used by the validator rewards monitor's Chain DAG update procedure. - The `getForkedBlock` procedure from the `blockchain_dag.nim` module is made public to be used by the validator rewards monitor's Chain DAG update procedure.
2022-01-17 12:58:33 +00:00
Implement all sync committee duties in the validator client (#3583) Other changes: * logtrace can now verify sync committee messages and contributions * Many unnecessary use of pairs() have been removed for consistency * Map 40x BN response codes to BeaconNodeStatus.Incompatible in the VC
2022-05-10 10:03:40 +00:00
for index, validator in info.validators:
More efficient reward data persistance; Address review comments The new format is based on compressed CSV files in two channels: * Detailed per-epoch data * Aggregated "daily" summaries The use of append-only CSV file speeds up significantly the epoch processing speed during data generation. The use of compression results in smaller storage requirements overall. The use of the aggregated files has a very minor cost in both CPU and storage, but leads to near interactive speed for report generation. Other changes: - Implemented support for graceful shut downs to avoid corrupting the saved files. - Fixed a memory leak caused by lacking `StateCache` clean up on each iteration. - Addressed review comments - Moved the rewards and penalties calculation code in a separate module Required invasive changes to existing modules: - The `data` field of the `KeyedBlockRef` type is made public to be used by the validator rewards monitor's Chain DAG update procedure. - The `getForkedBlock` procedure from the `blockchain_dag.nim` module is made public to be used by the validator rewards monitor's Chain DAG update procedure.
2022-01-17 12:58:33 +00:00
if not is_eligible_validator(validator):
continue
let base_reward = get_base_reward_sqrt(
state, index.ValidatorIndex, total_balance_sqrt)
make `Gwei` `distinct` (#6090) #6087 introduced a subtle change to `nim-web3` resulting in `Gwei` to be serialized differently than before. Using a `distinct` type for `Gwei` improves type safety and avoids such problems in the future.
2024-03-19 13:22:07 +00:00
template get_attestation_component_reward_helper(
attesting_balance: Gwei): Gwei =
More efficient reward data persistance; Address review comments The new format is based on compressed CSV files in two channels: * Detailed per-epoch data * Aggregated "daily" summaries The use of append-only CSV file speeds up significantly the epoch processing speed during data generation. The use of compression results in smaller storage requirements overall. The use of the aggregated files has a very minor cost in both CPU and storage, but leads to near interactive speed for report generation. Other changes: - Implemented support for graceful shut downs to avoid corrupting the saved files. - Fixed a memory leak caused by lacking `StateCache` clean up on each iteration. - Addressed review comments - Moved the rewards and penalties calculation code in a separate module Required invasive changes to existing modules: - The `data` field of the `KeyedBlockRef` type is made public to be used by the validator rewards monitor's Chain DAG update procedure. - The `getForkedBlock` procedure from the `blockchain_dag.nim` module is made public to be used by the validator rewards monitor's Chain DAG update procedure.
2022-01-17 12:58:33 +00:00
get_attestation_component_reward(attesting_balance,
make `Gwei` `distinct` (#6090) #6087 introduced a subtle change to `nim-web3` resulting in `Gwei` to be serialized differently than before. Using a `distinct` type for `Gwei` improves type safety and avoids such problems in the future.
2024-03-19 13:22:07 +00:00
info.balances.current_epoch, base_reward, finality_delay)
More efficient reward data persistance; Address review comments The new format is based on compressed CSV files in two channels: * Detailed per-epoch data * Aggregated "daily" summaries The use of append-only CSV file speeds up significantly the epoch processing speed during data generation. The use of compression results in smaller storage requirements overall. The use of the aggregated files has a very minor cost in both CPU and storage, but leads to near interactive speed for report generation. Other changes: - Implemented support for graceful shut downs to avoid corrupting the saved files. - Fixed a memory leak caused by lacking `StateCache` clean up on each iteration. - Addressed review comments - Moved the rewards and penalties calculation code in a separate module Required invasive changes to existing modules: - The `data` field of the `KeyedBlockRef` type is made public to be used by the validator rewards monitor's Chain DAG update procedure. - The `getForkedBlock` procedure from the `blockchain_dag.nim` module is made public to be used by the validator rewards monitor's Chain DAG update procedure.
2022-01-17 12:58:33 +00:00
template rp: untyped = rewardsAndPenalties[index]
rp.source_outcome = get_source_delta(
validator, base_reward, info.balances, finality_delay).getOutcome
rp.max_source_reward = get_attestation_component_reward_helper(
info.balances.previous_epoch_attesters)
rp.target_outcome = get_target_delta(
validator, base_reward, info.balances, finality_delay).getOutcome
rp.max_target_reward = get_attestation_component_reward_helper(
info.balances.previous_epoch_target_attesters)
rp.head_outcome = get_head_delta(
validator, base_reward, info.balances, finality_delay).getOutcome
rp.max_head_reward = get_attestation_component_reward_helper(
info.balances.previous_epoch_head_attesters)
let (inclusion_delay_delta, proposer_delta) = get_inclusion_delay_delta(
validator, base_reward)
rp.inclusion_delay_outcome = inclusion_delay_delta.getOutcome
rp.max_inclusion_delay_reward =
base_reward - state_transition_epoch.get_proposer_reward(base_reward)
rp.inactivity_penalty = get_inactivity_penalty_delta(
validator, base_reward, finality_delay).penalties
if proposer_delta.isSome:
let proposer_index = proposer_delta.get[0]
if proposer_index < info.validators.lenu64:
rewardsAndPenalties[proposer_index].proposer_outcome +=
proposer_delta.get[1].getOutcome
rewardsAndPenalties.collectSlashings(state, info.balances.current_epoch)
Fix a reward calculation bug affecting Prater epoch 64781 (#3428) To calculate the deltas correctly, the `process_inactivity_updates` function must be called before the rewards and penalties processing code in order to update the `inactivity_scores` field in the state. This would have required duplicating more logic from the spec in the ncli modules, so I've decided to pay the price of introducing a run-time copy of the state at each epoch which eliminates the need to duplicate logic (both for this fix and the previous one). Other changes: * Fixes for the read-only mode of the `BeaconChainDb` * Fix an uint64 underflow in the debug output procedure for printing balance deltas * Allow Bellatrix states in the reward computation helpers
2022-02-22 12:14:17 +00:00
proc collectEpochRewardsAndPenalties*(
More efficient reward data persistance; Address review comments The new format is based on compressed CSV files in two channels: * Detailed per-epoch data * Aggregated "daily" summaries The use of append-only CSV file speeds up significantly the epoch processing speed during data generation. The use of compression results in smaller storage requirements overall. The use of the aggregated files has a very minor cost in both CPU and storage, but leads to near interactive speed for report generation. Other changes: - Implemented support for graceful shut downs to avoid corrupting the saved files. - Fixed a memory leak caused by lacking `StateCache` clean up on each iteration. - Addressed review comments - Moved the rewards and penalties calculation code in a separate module Required invasive changes to existing modules: - The `data` field of the `KeyedBlockRef` type is made public to be used by the validator rewards monitor's Chain DAG update procedure. - The `getForkedBlock` procedure from the `blockchain_dag.nim` module is made public to be used by the validator rewards monitor's Chain DAG update procedure.
2022-01-17 12:58:33 +00:00
rewardsAndPenalties: var seq[RewardsAndPenalties],
fill some capella/EIP4844 missing implementations (#4585)
2023-02-02 22:24:06 +00:00
state: var (altair.BeaconState | bellatrix.BeaconState |
add Electra overloads for spec functions; add Electra block processing (#5963)
2024-02-26 02:38:21 +00:00
capella.BeaconState | deneb.BeaconState | electra.BeaconState),
Bugfix: Take into account the finalization delay in the ncli_db rewards calculation This fixes a reward calculation error affecting Prater's epoch 31256
2022-01-23 19:41:29 +00:00
cache: var StateCache, cfg: RuntimeConfig, flags: UpdateFlags) =
More efficient reward data persistance; Address review comments The new format is based on compressed CSV files in two channels: * Detailed per-epoch data * Aggregated "daily" summaries The use of append-only CSV file speeds up significantly the epoch processing speed during data generation. The use of compression results in smaller storage requirements overall. The use of the aggregated files has a very minor cost in both CPU and storage, but leads to near interactive speed for report generation. Other changes: - Implemented support for graceful shut downs to avoid corrupting the saved files. - Fixed a memory leak caused by lacking `StateCache` clean up on each iteration. - Addressed review comments - Moved the rewards and penalties calculation code in a separate module Required invasive changes to existing modules: - The `data` field of the `KeyedBlockRef` type is made public to be used by the validator rewards monitor's Chain DAG update procedure. - The `getForkedBlock` procedure from the `blockchain_dag.nim` module is made public to be used by the validator rewards monitor's Chain DAG update procedure.
2022-01-17 12:58:33 +00:00
if get_current_epoch(state) == GENESIS_EPOCH:
return
var info: altair.EpochInfo
info.init(state)
doAssert info.validators.len == state.validators.len
rewardsAndPenalties.setLen(state.validators.len)
Fix a reward calculation bug affecting Prater epoch 64781 (#3428) To calculate the deltas correctly, the `process_inactivity_updates` function must be called before the rewards and penalties processing code in order to update the `inactivity_scores` field in the state. This would have required duplicating more logic from the spec in the ncli modules, so I've decided to pay the price of introducing a run-time copy of the state at each epoch which eliminates the need to duplicate logic (both for this fix and the previous one). Other changes: * Fixes for the read-only mode of the `BeaconChainDb` * Fix an uint64 underflow in the debug output procedure for printing balance deltas * Allow Bellatrix states in the reward computation helpers
2022-02-22 12:14:17 +00:00
process_justification_and_finalization(state, info.balances, flags)
process_inactivity_updates(cfg, state, info)
More efficient reward data persistance; Address review comments The new format is based on compressed CSV files in two channels: * Detailed per-epoch data * Aggregated "daily" summaries The use of append-only CSV file speeds up significantly the epoch processing speed during data generation. The use of compression results in smaller storage requirements overall. The use of the aggregated files has a very minor cost in both CPU and storage, but leads to near interactive speed for report generation. Other changes: - Implemented support for graceful shut downs to avoid corrupting the saved files. - Fixed a memory leak caused by lacking `StateCache` clean up on each iteration. - Addressed review comments - Moved the rewards and penalties calculation code in a separate module Required invasive changes to existing modules: - The `data` field of the `KeyedBlockRef` type is made public to be used by the validator rewards monitor's Chain DAG update procedure. - The `getForkedBlock` procedure from the `blockchain_dag.nim` module is made public to be used by the validator rewards monitor's Chain DAG update procedure.
2022-01-17 12:58:33 +00:00
let
total_active_balance = info.balances.current_epoch
base_reward_per_increment = get_base_reward_per_increment(
total_active_balance)
Fix a reward calculation bug affecting Prater epoch 64781 (#3428) To calculate the deltas correctly, the `process_inactivity_updates` function must be called before the rewards and penalties processing code in order to update the `inactivity_scores` field in the state. This would have required duplicating more logic from the spec in the ncli modules, so I've decided to pay the price of introducing a run-time copy of the state at each epoch which eliminates the need to duplicate logic (both for this fix and the previous one). Other changes: * Fixes for the read-only mode of the `BeaconChainDb` * Fix an uint64 underflow in the debug output procedure for printing balance deltas * Allow Bellatrix states in the reward computation helpers
2022-02-22 12:14:17 +00:00
finality_delay = get_finality_delay(state)
More efficient reward data persistance; Address review comments The new format is based on compressed CSV files in two channels: * Detailed per-epoch data * Aggregated "daily" summaries The use of append-only CSV file speeds up significantly the epoch processing speed during data generation. The use of compression results in smaller storage requirements overall. The use of the aggregated files has a very minor cost in both CPU and storage, but leads to near interactive speed for report generation. Other changes: - Implemented support for graceful shut downs to avoid corrupting the saved files. - Fixed a memory leak caused by lacking `StateCache` clean up on each iteration. - Addressed review comments - Moved the rewards and penalties calculation code in a separate module Required invasive changes to existing modules: - The `data` field of the `KeyedBlockRef` type is made public to be used by the validator rewards monitor's Chain DAG update procedure. - The `getForkedBlock` procedure from the `blockchain_dag.nim` module is made public to be used by the validator rewards monitor's Chain DAG update procedure.
2022-01-17 12:58:33 +00:00
restore full test coverage for process_rewards_and_penalties (#5407) * restore full test coverage for process_rewards_and_penalties * adjust ncli_db to use new iterator
2023-09-08 16:42:18 +00:00
for validator_index, reward_source, reward_target, reward_head,
penalty_source, penalty_target, penalty_inactivity in
get_flag_and_inactivity_deltas(
cfg, state, base_reward_per_increment, info, finality_delay):
template rp: untyped = rewardsAndPenalties[validator_index]
let
base_reward = get_base_reward_increment(
state, validator_index, base_reward_per_increment)
active_increments = get_active_increments(info)
template unslashed_participating_increment(flag_index: untyped): untyped =
get_unslashed_participating_increment(info, flag_index)
template max_flag_index_reward(flag_index: untyped): untyped =
get_flag_index_reward(
state, base_reward, active_increments,
unslashed_participating_increment(flag_index),
PARTICIPATION_FLAG_WEIGHTS[flag_index], finality_delay)
rp.source_outcome = reward_source.int64 - penalty_source.int64
rp.max_source_reward =
max_flag_index_reward(TimelyFlag.TIMELY_SOURCE_FLAG_INDEX)
rp.target_outcome = reward_target.int64 - penalty_target.int64
rp.max_target_reward =
max_flag_index_reward(TimelyFlag.TIMELY_TARGET_FLAG_INDEX)
rp.head_outcome = reward_head.int64
rp.max_head_reward =
max_flag_index_reward(TimelyFlag.TIMELY_HEAD_FLAG_INDEX)
rewardsAndPenalties[validator_index].inactivity_penalty +=
penalty_inactivity
More efficient reward data persistance; Address review comments The new format is based on compressed CSV files in two channels: * Detailed per-epoch data * Aggregated "daily" summaries The use of append-only CSV file speeds up significantly the epoch processing speed during data generation. The use of compression results in smaller storage requirements overall. The use of the aggregated files has a very minor cost in both CPU and storage, but leads to near interactive speed for report generation. Other changes: - Implemented support for graceful shut downs to avoid corrupting the saved files. - Fixed a memory leak caused by lacking `StateCache` clean up on each iteration. - Addressed review comments - Moved the rewards and penalties calculation code in a separate module Required invasive changes to existing modules: - The `data` field of the `KeyedBlockRef` type is made public to be used by the validator rewards monitor's Chain DAG update procedure. - The `getForkedBlock` procedure from the `blockchain_dag.nim` module is made public to be used by the validator rewards monitor's Chain DAG update procedure.
2022-01-17 12:58:33 +00:00
rewardsAndPenalties.collectSlashings(state, info.balances.current_epoch)
proc -> func, mainly in spec/state transition and adjecent modules (#3405)
2022-02-17 11:53:55 +00:00
func collectFromSlashedValidator(
More efficient reward data persistance; Address review comments The new format is based on compressed CSV files in two channels: * Detailed per-epoch data * Aggregated "daily" summaries The use of append-only CSV file speeds up significantly the epoch processing speed during data generation. The use of compression results in smaller storage requirements overall. The use of the aggregated files has a very minor cost in both CPU and storage, but leads to near interactive speed for report generation. Other changes: - Implemented support for graceful shut downs to avoid corrupting the saved files. - Fixed a memory leak caused by lacking `StateCache` clean up on each iteration. - Addressed review comments - Moved the rewards and penalties calculation code in a separate module Required invasive changes to existing modules: - The `data` field of the `KeyedBlockRef` type is made public to be used by the validator rewards monitor's Chain DAG update procedure. - The `getForkedBlock` procedure from the `blockchain_dag.nim` module is made public to be used by the validator rewards monitor's Chain DAG update procedure.
2022-01-17 12:58:33 +00:00
rewardsAndPenalties: var seq[RewardsAndPenalties],
state: ForkyBeaconState, slashedIndex, proposerIndex: ValidatorIndex) =
template slashed_validator: untyped = state.validators[slashedIndex]
add EF consensus spec test attester slashing, proposer slashing, voluntary exit, and execution layer withdrawal request tests (#6234)
2024-04-25 08:41:17 +00:00
let
slashingPenalty =
get_slashing_penalty(state, slashed_validator.effective_balance)
whistleblowerReward =
get_whistleblower_reward(state, slashed_validator.effective_balance)
More efficient reward data persistance; Address review comments The new format is based on compressed CSV files in two channels: * Detailed per-epoch data * Aggregated "daily" summaries The use of append-only CSV file speeds up significantly the epoch processing speed during data generation. The use of compression results in smaller storage requirements overall. The use of the aggregated files has a very minor cost in both CPU and storage, but leads to near interactive speed for report generation. Other changes: - Implemented support for graceful shut downs to avoid corrupting the saved files. - Fixed a memory leak caused by lacking `StateCache` clean up on each iteration. - Addressed review comments - Moved the rewards and penalties calculation code in a separate module Required invasive changes to existing modules: - The `data` field of the `KeyedBlockRef` type is made public to be used by the validator rewards monitor's Chain DAG update procedure. - The `getForkedBlock` procedure from the `blockchain_dag.nim` module is made public to be used by the validator rewards monitor's Chain DAG update procedure.
2022-01-17 12:58:33 +00:00
rewardsAndPenalties[slashedIndex].slashing_outcome -= slashingPenalty.int64
rewardsAndPenalties[proposerIndex].slashing_outcome += whistleblowerReward.int64
proc -> func, mainly in spec/state transition and adjecent modules (#3405)
2022-02-17 11:53:55 +00:00
func collectFromProposerSlashings(
More efficient reward data persistance; Address review comments The new format is based on compressed CSV files in two channels: * Detailed per-epoch data * Aggregated "daily" summaries The use of append-only CSV file speeds up significantly the epoch processing speed during data generation. The use of compression results in smaller storage requirements overall. The use of the aggregated files has a very minor cost in both CPU and storage, but leads to near interactive speed for report generation. Other changes: - Implemented support for graceful shut downs to avoid corrupting the saved files. - Fixed a memory leak caused by lacking `StateCache` clean up on each iteration. - Addressed review comments - Moved the rewards and penalties calculation code in a separate module Required invasive changes to existing modules: - The `data` field of the `KeyedBlockRef` type is made public to be used by the validator rewards monitor's Chain DAG update procedure. - The `getForkedBlock` procedure from the `blockchain_dag.nim` module is made public to be used by the validator rewards monitor's Chain DAG update procedure.
2022-01-17 12:58:33 +00:00
rewardsAndPenalties: var seq[RewardsAndPenalties],
forkedState: ForkedHashedBeaconState,
forkedBlock: ForkedTrustedSignedBeaconBlock) =
withStateAndBlck(forkedState, forkedBlock):
`blck` --> `forkyBlck` when using `withBlck` / `withStateAndBlck` (#5451) For symmetry with `forkyState` when using `withState`, and to avoid problems with shadowing of `blck` when using `withBlck` in `template`, also rename the injected `blck` to `forkyBlck`. - https://github.com/nim-lang/Nim/issues/22698
2023-09-21 10:49:14 +00:00
for proposer_slashing in forkyBlck.message.body.proposer_slashings:
switch `withStateAndBlck` usage to `forkyState`; rm unused `tests/mocking/` modules (#4206)
2022-10-03 13:08:50 +00:00
doAssert check_proposer_slashing(
forkyState.data, proposer_slashing, {}).isOk
let slashedIndex =
proposer_slashing.signed_header_1.message.proposer_index
rewardsAndPenalties.collectFromSlashedValidator(
forkyState.data, slashedIndex.ValidatorIndex,
`blck` --> `forkyBlck` when using `withBlck` / `withStateAndBlck` (#5451) For symmetry with `forkyState` when using `withState`, and to avoid problems with shadowing of `blck` when using `withBlck` in `template`, also rename the injected `blck` to `forkyBlck`. - https://github.com/nim-lang/Nim/issues/22698
2023-09-21 10:49:14 +00:00
forkyBlck.message.proposer_index.ValidatorIndex)
More efficient reward data persistance; Address review comments The new format is based on compressed CSV files in two channels: * Detailed per-epoch data * Aggregated "daily" summaries The use of append-only CSV file speeds up significantly the epoch processing speed during data generation. The use of compression results in smaller storage requirements overall. The use of the aggregated files has a very minor cost in both CPU and storage, but leads to near interactive speed for report generation. Other changes: - Implemented support for graceful shut downs to avoid corrupting the saved files. - Fixed a memory leak caused by lacking `StateCache` clean up on each iteration. - Addressed review comments - Moved the rewards and penalties calculation code in a separate module Required invasive changes to existing modules: - The `data` field of the `KeyedBlockRef` type is made public to be used by the validator rewards monitor's Chain DAG update procedure. - The `getForkedBlock` procedure from the `blockchain_dag.nim` module is made public to be used by the validator rewards monitor's Chain DAG update procedure.
2022-01-17 12:58:33 +00:00
proc -> func, mainly in spec/state transition and adjecent modules (#3405)
2022-02-17 11:53:55 +00:00
func collectFromAttesterSlashings(
More efficient reward data persistance; Address review comments The new format is based on compressed CSV files in two channels: * Detailed per-epoch data * Aggregated "daily" summaries The use of append-only CSV file speeds up significantly the epoch processing speed during data generation. The use of compression results in smaller storage requirements overall. The use of the aggregated files has a very minor cost in both CPU and storage, but leads to near interactive speed for report generation. Other changes: - Implemented support for graceful shut downs to avoid corrupting the saved files. - Fixed a memory leak caused by lacking `StateCache` clean up on each iteration. - Addressed review comments - Moved the rewards and penalties calculation code in a separate module Required invasive changes to existing modules: - The `data` field of the `KeyedBlockRef` type is made public to be used by the validator rewards monitor's Chain DAG update procedure. - The `getForkedBlock` procedure from the `blockchain_dag.nim` module is made public to be used by the validator rewards monitor's Chain DAG update procedure.
2022-01-17 12:58:33 +00:00
rewardsAndPenalties: var seq[RewardsAndPenalties],
forkedState: ForkedHashedBeaconState,
forkedBlock: ForkedTrustedSignedBeaconBlock) =
withStateAndBlck(forkedState, forkedBlock):
`blck` --> `forkyBlck` when using `withBlck` / `withStateAndBlck` (#5451) For symmetry with `forkyState` when using `withState`, and to avoid problems with shadowing of `blck` when using `withBlck` in `template`, also rename the injected `blck` to `forkyBlck`. - https://github.com/nim-lang/Nim/issues/22698
2023-09-21 10:49:14 +00:00
for attester_slashing in forkyBlck.message.body.attester_slashings:
More efficient reward data persistance; Address review comments The new format is based on compressed CSV files in two channels: * Detailed per-epoch data * Aggregated "daily" summaries The use of append-only CSV file speeds up significantly the epoch processing speed during data generation. The use of compression results in smaller storage requirements overall. The use of the aggregated files has a very minor cost in both CPU and storage, but leads to near interactive speed for report generation. Other changes: - Implemented support for graceful shut downs to avoid corrupting the saved files. - Fixed a memory leak caused by lacking `StateCache` clean up on each iteration. - Addressed review comments - Moved the rewards and penalties calculation code in a separate module Required invasive changes to existing modules: - The `data` field of the `KeyedBlockRef` type is made public to be used by the validator rewards monitor's Chain DAG update procedure. - The `getForkedBlock` procedure from the `blockchain_dag.nim` module is made public to be used by the validator rewards monitor's Chain DAG update procedure.
2022-01-17 12:58:33 +00:00
let attester_slashing_validity = check_attester_slashing(
switch `withStateAndBlck` usage to `forkyState`; rm unused `tests/mocking/` modules (#4206)
2022-10-03 13:08:50 +00:00
forkyState.data, attester_slashing, {})
More efficient reward data persistance; Address review comments The new format is based on compressed CSV files in two channels: * Detailed per-epoch data * Aggregated "daily" summaries The use of append-only CSV file speeds up significantly the epoch processing speed during data generation. The use of compression results in smaller storage requirements overall. The use of the aggregated files has a very minor cost in both CPU and storage, but leads to near interactive speed for report generation. Other changes: - Implemented support for graceful shut downs to avoid corrupting the saved files. - Fixed a memory leak caused by lacking `StateCache` clean up on each iteration. - Addressed review comments - Moved the rewards and penalties calculation code in a separate module Required invasive changes to existing modules: - The `data` field of the `KeyedBlockRef` type is made public to be used by the validator rewards monitor's Chain DAG update procedure. - The `getForkedBlock` procedure from the `blockchain_dag.nim` module is made public to be used by the validator rewards monitor's Chain DAG update procedure.
2022-01-17 12:58:33 +00:00
doAssert attester_slashing_validity.isOk
for slashedIndex in attester_slashing_validity.value:
rewardsAndPenalties.collectFromSlashedValidator(
switch `withStateAndBlck` usage to `forkyState`; rm unused `tests/mocking/` modules (#4206)
2022-10-03 13:08:50 +00:00
forkyState.data, slashedIndex,
`blck` --> `forkyBlck` when using `withBlck` / `withStateAndBlck` (#5451) For symmetry with `forkyState` when using `withState`, and to avoid problems with shadowing of `blck` when using `withBlck` in `template`, also rename the injected `blck` to `forkyBlck`. - https://github.com/nim-lang/Nim/issues/22698
2023-09-21 10:49:14 +00:00
forkyBlck.message.proposer_index.ValidatorIndex)
More efficient reward data persistance; Address review comments The new format is based on compressed CSV files in two channels: * Detailed per-epoch data * Aggregated "daily" summaries The use of append-only CSV file speeds up significantly the epoch processing speed during data generation. The use of compression results in smaller storage requirements overall. The use of the aggregated files has a very minor cost in both CPU and storage, but leads to near interactive speed for report generation. Other changes: - Implemented support for graceful shut downs to avoid corrupting the saved files. - Fixed a memory leak caused by lacking `StateCache` clean up on each iteration. - Addressed review comments - Moved the rewards and penalties calculation code in a separate module Required invasive changes to existing modules: - The `data` field of the `KeyedBlockRef` type is made public to be used by the validator rewards monitor's Chain DAG update procedure. - The `getForkedBlock` procedure from the `blockchain_dag.nim` module is made public to be used by the validator rewards monitor's Chain DAG update procedure.
2022-01-17 12:58:33 +00:00
proc -> func, mainly in spec/state transition and adjecent modules (#3405)
2022-02-17 11:53:55 +00:00
func collectFromAttestations(
More efficient reward data persistance; Address review comments The new format is based on compressed CSV files in two channels: * Detailed per-epoch data * Aggregated "daily" summaries The use of append-only CSV file speeds up significantly the epoch processing speed during data generation. The use of compression results in smaller storage requirements overall. The use of the aggregated files has a very minor cost in both CPU and storage, but leads to near interactive speed for report generation. Other changes: - Implemented support for graceful shut downs to avoid corrupting the saved files. - Fixed a memory leak caused by lacking `StateCache` clean up on each iteration. - Addressed review comments - Moved the rewards and penalties calculation code in a separate module Required invasive changes to existing modules: - The `data` field of the `KeyedBlockRef` type is made public to be used by the validator rewards monitor's Chain DAG update procedure. - The `getForkedBlock` procedure from the `blockchain_dag.nim` module is made public to be used by the validator rewards monitor's Chain DAG update procedure.
2022-01-17 12:58:33 +00:00
rewardsAndPenalties: var seq[RewardsAndPenalties],
forkedState: ForkedHashedBeaconState,
forkedBlock: ForkedTrustedSignedBeaconBlock,
epochParticipationFlags: var ParticipationFlags,
cache: var StateCache) =
withStateAndBlck(forkedState, forkedBlock):
rename `stateFork` > `consensusFork` (#4718) Just the variable, not yet `lcDataForkAtStateFork` / `atStateFork`. - Shorten comment in `light_client.nim` to keep line width - Do not rename `stateFork` mention in `runProposalForkchoiceUpdated`. - Do not rename `stateFork` in `getStateField(dag.headState, fork)` Rest is just a mechanical mass replace
2023-03-11 00:35:52 +00:00
when consensusFork > ConsensusFork.Phase0:
More efficient reward data persistance; Address review comments The new format is based on compressed CSV files in two channels: * Detailed per-epoch data * Aggregated "daily" summaries The use of append-only CSV file speeds up significantly the epoch processing speed during data generation. The use of compression results in smaller storage requirements overall. The use of the aggregated files has a very minor cost in both CPU and storage, but leads to near interactive speed for report generation. Other changes: - Implemented support for graceful shut downs to avoid corrupting the saved files. - Fixed a memory leak caused by lacking `StateCache` clean up on each iteration. - Addressed review comments - Moved the rewards and penalties calculation code in a separate module Required invasive changes to existing modules: - The `data` field of the `KeyedBlockRef` type is made public to be used by the validator rewards monitor's Chain DAG update procedure. - The `getForkedBlock` procedure from the `blockchain_dag.nim` module is made public to be used by the validator rewards monitor's Chain DAG update procedure.
2022-01-17 12:58:33 +00:00
let base_reward_per_increment = get_base_reward_per_increment(
switch `withStateAndBlck` usage to `forkyState`; rm unused `tests/mocking/` modules (#4206)
2022-10-03 13:08:50 +00:00
get_total_active_balance(forkyState.data, cache))
make `Gwei` `distinct` (#6090) #6087 introduced a subtle change to `nim-web3` resulting in `Gwei` to be serialized differently than before. Using a `distinct` type for `Gwei` improves type safety and avoids such problems in the future.
2024-03-19 13:22:07 +00:00
doAssert base_reward_per_increment > 0.Gwei
`blck` --> `forkyBlck` when using `withBlck` / `withStateAndBlck` (#5451) For symmetry with `forkyState` when using `withState`, and to avoid problems with shadowing of `blck` when using `withBlck` in `template`, also rename the injected `blck` to `forkyBlck`. - https://github.com/nim-lang/Nim/issues/22698
2023-09-21 10:49:14 +00:00
for attestation in forkyBlck.message.body.attestations:
switch `withStateAndBlck` usage to `forkyState`; rm unused `tests/mocking/` modules (#4206)
2022-10-03 13:08:50 +00:00
doAssert check_attestation(
electra attestation updates (#6295) * electra attestation updates In Electra, we have two attestation formats: on-chain and on-network - the former combines all committees of a slot in a single committee bit list. This PR makes a number of cleanups to move towards fixing this - attestation packing however still needs to be fixed as it currently creates attestations with a single committee only which is very inefficient. * more attestations in the blocks * signing and aggregation fixes * tool fix * test, import
2024-05-17 12:37:41 +00:00
forkyState.data, attestation, {}, cache, true).isOk
More efficient reward data persistance; Address review comments The new format is based on compressed CSV files in two channels: * Detailed per-epoch data * Aggregated "daily" summaries The use of append-only CSV file speeds up significantly the epoch processing speed during data generation. The use of compression results in smaller storage requirements overall. The use of the aggregated files has a very minor cost in both CPU and storage, but leads to near interactive speed for report generation. Other changes: - Implemented support for graceful shut downs to avoid corrupting the saved files. - Fixed a memory leak caused by lacking `StateCache` clean up on each iteration. - Addressed review comments - Moved the rewards and penalties calculation code in a separate module Required invasive changes to existing modules: - The `data` field of the `KeyedBlockRef` type is made public to be used by the validator rewards monitor's Chain DAG update procedure. - The `getForkedBlock` procedure from the `blockchain_dag.nim` module is made public to be used by the validator rewards monitor's Chain DAG update procedure.
2022-01-17 12:58:33 +00:00
let proposerReward =
switch `withStateAndBlck` usage to `forkyState`; rm unused `tests/mocking/` modules (#4206)
2022-10-03 13:08:50 +00:00
if attestation.data.target.epoch == get_current_epoch(forkyState.data):
More efficient reward data persistance; Address review comments The new format is based on compressed CSV files in two channels: * Detailed per-epoch data * Aggregated "daily" summaries The use of append-only CSV file speeds up significantly the epoch processing speed during data generation. The use of compression results in smaller storage requirements overall. The use of the aggregated files has a very minor cost in both CPU and storage, but leads to near interactive speed for report generation. Other changes: - Implemented support for graceful shut downs to avoid corrupting the saved files. - Fixed a memory leak caused by lacking `StateCache` clean up on each iteration. - Addressed review comments - Moved the rewards and penalties calculation code in a separate module Required invasive changes to existing modules: - The `data` field of the `KeyedBlockRef` type is made public to be used by the validator rewards monitor's Chain DAG update procedure. - The `getForkedBlock` procedure from the `blockchain_dag.nim` module is made public to be used by the validator rewards monitor's Chain DAG update procedure.
2022-01-17 12:58:33 +00:00
get_proposer_reward(
switch `withStateAndBlck` usage to `forkyState`; rm unused `tests/mocking/` modules (#4206)
2022-10-03 13:08:50 +00:00
forkyState.data, attestation, base_reward_per_increment, cache,
More efficient reward data persistance; Address review comments The new format is based on compressed CSV files in two channels: * Detailed per-epoch data * Aggregated "daily" summaries The use of append-only CSV file speeds up significantly the epoch processing speed during data generation. The use of compression results in smaller storage requirements overall. The use of the aggregated files has a very minor cost in both CPU and storage, but leads to near interactive speed for report generation. Other changes: - Implemented support for graceful shut downs to avoid corrupting the saved files. - Fixed a memory leak caused by lacking `StateCache` clean up on each iteration. - Addressed review comments - Moved the rewards and penalties calculation code in a separate module Required invasive changes to existing modules: - The `data` field of the `KeyedBlockRef` type is made public to be used by the validator rewards monitor's Chain DAG update procedure. - The `getForkedBlock` procedure from the `blockchain_dag.nim` module is made public to be used by the validator rewards monitor's Chain DAG update procedure.
2022-01-17 12:58:33 +00:00
epochParticipationFlags.currentEpochParticipation)
else:
get_proposer_reward(
switch `withStateAndBlck` usage to `forkyState`; rm unused `tests/mocking/` modules (#4206)
2022-10-03 13:08:50 +00:00
forkyState.data, attestation, base_reward_per_increment, cache,
More efficient reward data persistance; Address review comments The new format is based on compressed CSV files in two channels: * Detailed per-epoch data * Aggregated "daily" summaries The use of append-only CSV file speeds up significantly the epoch processing speed during data generation. The use of compression results in smaller storage requirements overall. The use of the aggregated files has a very minor cost in both CPU and storage, but leads to near interactive speed for report generation. Other changes: - Implemented support for graceful shut downs to avoid corrupting the saved files. - Fixed a memory leak caused by lacking `StateCache` clean up on each iteration. - Addressed review comments - Moved the rewards and penalties calculation code in a separate module Required invasive changes to existing modules: - The `data` field of the `KeyedBlockRef` type is made public to be used by the validator rewards monitor's Chain DAG update procedure. - The `getForkedBlock` procedure from the `blockchain_dag.nim` module is made public to be used by the validator rewards monitor's Chain DAG update procedure.
2022-01-17 12:58:33 +00:00
epochParticipationFlags.previousEpochParticipation)
`blck` --> `forkyBlck` when using `withBlck` / `withStateAndBlck` (#5451) For symmetry with `forkyState` when using `withState`, and to avoid problems with shadowing of `blck` when using `withBlck` in `template`, also rename the injected `blck` to `forkyBlck`. - https://github.com/nim-lang/Nim/issues/22698
2023-09-21 10:49:14 +00:00
rewardsAndPenalties[forkyBlck.message.proposer_index]
.proposer_outcome += proposerReward.int64
switch `withStateAndBlck` usage to `forkyState`; rm unused `tests/mocking/` modules (#4206)
2022-10-03 13:08:50 +00:00
let inclusionDelay = forkyState.data.slot - attestation.data.slot
add EF consensus spec test Electra attestation operations fixture (#6248)
2024-04-28 00:52:14 +00:00
when consensusFork >= ConsensusFork.Electra:
for index in get_attesting_indices(
forkyState.data, attestation.data, attestation.aggregation_bits,
attestation.committee_bits, cache):
rewardsAndPenalties[index].inclusion_delay =
some(inclusionDelay.uint64)
else:
for index in get_attesting_indices(
forkyState.data, attestation.data, attestation.aggregation_bits,
cache):
rewardsAndPenalties[index].inclusion_delay =
some(inclusionDelay.uint64)
More efficient reward data persistance; Address review comments The new format is based on compressed CSV files in two channels: * Detailed per-epoch data * Aggregated "daily" summaries The use of append-only CSV file speeds up significantly the epoch processing speed during data generation. The use of compression results in smaller storage requirements overall. The use of the aggregated files has a very minor cost in both CPU and storage, but leads to near interactive speed for report generation. Other changes: - Implemented support for graceful shut downs to avoid corrupting the saved files. - Fixed a memory leak caused by lacking `StateCache` clean up on each iteration. - Addressed review comments - Moved the rewards and penalties calculation code in a separate module Required invasive changes to existing modules: - The `data` field of the `KeyedBlockRef` type is made public to be used by the validator rewards monitor's Chain DAG update procedure. - The `getForkedBlock` procedure from the `blockchain_dag.nim` module is made public to be used by the validator rewards monitor's Chain DAG update procedure.
2022-01-17 12:58:33 +00:00
replace validator Bloom filter with partial bucket sort (#6469)
2024-08-02 05:30:22 +00:00
from ".."/beacon_chain/validator_bucket_sort import
findValidatorIndex, sortValidatorBuckets
More efficient reward data persistance; Address review comments The new format is based on compressed CSV files in two channels: * Detailed per-epoch data * Aggregated "daily" summaries The use of append-only CSV file speeds up significantly the epoch processing speed during data generation. The use of compression results in smaller storage requirements overall. The use of the aggregated files has a very minor cost in both CPU and storage, but leads to near interactive speed for report generation. Other changes: - Implemented support for graceful shut downs to avoid corrupting the saved files. - Fixed a memory leak caused by lacking `StateCache` clean up on each iteration. - Addressed review comments - Moved the rewards and penalties calculation code in a separate module Required invasive changes to existing modules: - The `data` field of the `KeyedBlockRef` type is made public to be used by the validator rewards monitor's Chain DAG update procedure. - The `getForkedBlock` procedure from the `blockchain_dag.nim` module is made public to be used by the validator rewards monitor's Chain DAG update procedure.
2022-01-17 12:58:33 +00:00
proc collectFromDeposits(
rewardsAndPenalties: var seq[RewardsAndPenalties],
forkedState: ForkedHashedBeaconState,
forkedBlock: ForkedTrustedSignedBeaconBlock,
pubkeyToIndex: var PubkeyToIndexTable,
cfg: RuntimeConfig) =
withStateAndBlck(forkedState, forkedBlock):
`blck` --> `forkyBlck` when using `withBlck` / `withStateAndBlck` (#5451) For symmetry with `forkyState` when using `withState`, and to avoid problems with shadowing of `blck` when using `withBlck` in `template`, also rename the injected `blck` to `forkyBlck`. - https://github.com/nim-lang/Nim/issues/22698
2023-09-21 10:49:14 +00:00
for deposit in forkyBlck.message.body.deposits:
replace validator Bloom filter with partial bucket sort (#6469)
2024-08-02 05:30:22 +00:00
let
pubkey = deposit.data.pubkey
amount = deposit.data.amount
var index = findValidatorIndex(
forkyState.data.validators.asSeq, sortValidatorBuckets(
forkyState.data.validators.asSeq)[], pubkey)
document and clean up `ValidatorIndex` usage (#3651) * document static vs dynamic range checking requirements * add `vindices` iterator to iterate over valid validator indices in a state * clean up spec comments in general * fixup Co-authored-by: tersec <tersec@users.noreply.github.com>
2022-05-23 23:39:08 +00:00
if index.isNone:
if pubkey in pubkeyToIndex:
extend `ncli` exception handling (#5905) Make raised exceptions explicit in `ncli_common.nim`, and handle more of them in `ncli_db.nim` to have better UX when directories cannot be read or file names do not parse against the expected format.
2024-02-19 09:56:19 +00:00
try:
index = Opt[ValidatorIndex].ok(pubkeyToIndex[pubkey])
except KeyError as e:
raiseAssert "pubkey was checked to exist: " & e.msg
document and clean up `ValidatorIndex` usage (#3651) * document static vs dynamic range checking requirements * add `vindices` iterator to iterate over valid validator indices in a state * clean up spec comments in general * fixup Co-authored-by: tersec <tersec@users.noreply.github.com>
2022-05-23 23:39:08 +00:00
if index.isSome:
extend `ncli` exception handling (#5905) Make raised exceptions explicit in `ncli_common.nim`, and handle more of them in `ncli_db.nim` to have better UX when directories cannot be read or file names do not parse against the expected format.
2024-02-19 09:56:19 +00:00
try:
rewardsAndPenalties[index.get()].deposits += amount
except KeyError as e:
raiseAssert "rewardsAndPenalties lacks expected index " & $index.get()
More efficient reward data persistance; Address review comments The new format is based on compressed CSV files in two channels: * Detailed per-epoch data * Aggregated "daily" summaries The use of append-only CSV file speeds up significantly the epoch processing speed during data generation. The use of compression results in smaller storage requirements overall. The use of the aggregated files has a very minor cost in both CPU and storage, but leads to near interactive speed for report generation. Other changes: - Implemented support for graceful shut downs to avoid corrupting the saved files. - Fixed a memory leak caused by lacking `StateCache` clean up on each iteration. - Addressed review comments - Moved the rewards and penalties calculation code in a separate module Required invasive changes to existing modules: - The `data` field of the `KeyedBlockRef` type is made public to be used by the validator rewards monitor's Chain DAG update procedure. - The `getForkedBlock` procedure from the `blockchain_dag.nim` module is made public to be used by the validator rewards monitor's Chain DAG update procedure.
2022-01-17 12:58:33 +00:00
elif verify_deposit_signature(cfg, deposit.data):
document and clean up `ValidatorIndex` usage (#3651) * document static vs dynamic range checking requirements * add `vindices` iterator to iterate over valid validator indices in a state * clean up spec comments in general * fixup Co-authored-by: tersec <tersec@users.noreply.github.com>
2022-05-23 23:39:08 +00:00
pubkeyToIndex[pubkey] = ValidatorIndex(rewardsAndPenalties.len)
More efficient reward data persistance; Address review comments The new format is based on compressed CSV files in two channels: * Detailed per-epoch data * Aggregated "daily" summaries The use of append-only CSV file speeds up significantly the epoch processing speed during data generation. The use of compression results in smaller storage requirements overall. The use of the aggregated files has a very minor cost in both CPU and storage, but leads to near interactive speed for report generation. Other changes: - Implemented support for graceful shut downs to avoid corrupting the saved files. - Fixed a memory leak caused by lacking `StateCache` clean up on each iteration. - Addressed review comments - Moved the rewards and penalties calculation code in a separate module Required invasive changes to existing modules: - The `data` field of the `KeyedBlockRef` type is made public to be used by the validator rewards monitor's Chain DAG update procedure. - The `getForkedBlock` procedure from the `blockchain_dag.nim` module is made public to be used by the validator rewards monitor's Chain DAG update procedure.
2022-01-17 12:58:33 +00:00
rewardsAndPenalties.add(
RewardsAndPenalties(deposits: amount))
proc -> func, mainly in spec/state transition and adjecent modules (#3405)
2022-02-17 11:53:55 +00:00
func collectFromSyncAggregate(
More efficient reward data persistance; Address review comments The new format is based on compressed CSV files in two channels: * Detailed per-epoch data * Aggregated "daily" summaries The use of append-only CSV file speeds up significantly the epoch processing speed during data generation. The use of compression results in smaller storage requirements overall. The use of the aggregated files has a very minor cost in both CPU and storage, but leads to near interactive speed for report generation. Other changes: - Implemented support for graceful shut downs to avoid corrupting the saved files. - Fixed a memory leak caused by lacking `StateCache` clean up on each iteration. - Addressed review comments - Moved the rewards and penalties calculation code in a separate module Required invasive changes to existing modules: - The `data` field of the `KeyedBlockRef` type is made public to be used by the validator rewards monitor's Chain DAG update procedure. - The `getForkedBlock` procedure from the `blockchain_dag.nim` module is made public to be used by the validator rewards monitor's Chain DAG update procedure.
2022-01-17 12:58:33 +00:00
rewardsAndPenalties: var seq[RewardsAndPenalties],
forkedState: ForkedHashedBeaconState,
forkedBlock: ForkedTrustedSignedBeaconBlock,
cache: var StateCache) =
withStateAndBlck(forkedState, forkedBlock):
rename `stateFork` > `consensusFork` (#4718) Just the variable, not yet `lcDataForkAtStateFork` / `atStateFork`. - Shorten comment in `light_client.nim` to keep line width - Do not rename `stateFork` mention in `runProposalForkchoiceUpdated`. - Do not rename `stateFork` in `getStateField(dag.headState, fork)` Rest is just a mechanical mass replace
2023-03-11 00:35:52 +00:00
when consensusFork > ConsensusFork.Phase0:
switch `withStateAndBlck` usage to `forkyState`; rm unused `tests/mocking/` modules (#4206)
2022-10-03 13:08:50 +00:00
let
total_active_balance = get_total_active_balance(forkyState.data, cache)
participant_reward = get_participant_reward(total_active_balance)
proposer_reward =
state_transition_block.get_proposer_reward(participant_reward)
indices = get_sync_committee_cache(
forkyState.data, cache).current_sync_committee
More efficient reward data persistance; Address review comments The new format is based on compressed CSV files in two channels: * Detailed per-epoch data * Aggregated "daily" summaries The use of append-only CSV file speeds up significantly the epoch processing speed during data generation. The use of compression results in smaller storage requirements overall. The use of the aggregated files has a very minor cost in both CPU and storage, but leads to near interactive speed for report generation. Other changes: - Implemented support for graceful shut downs to avoid corrupting the saved files. - Fixed a memory leak caused by lacking `StateCache` clean up on each iteration. - Addressed review comments - Moved the rewards and penalties calculation code in a separate module Required invasive changes to existing modules: - The `data` field of the `KeyedBlockRef` type is made public to be used by the validator rewards monitor's Chain DAG update procedure. - The `getForkedBlock` procedure from the `blockchain_dag.nim` module is made public to be used by the validator rewards monitor's Chain DAG update procedure.
2022-01-17 12:58:33 +00:00
`blck` --> `forkyBlck` when using `withBlck` / `withStateAndBlck` (#5451) For symmetry with `forkyState` when using `withState`, and to avoid problems with shadowing of `blck` when using `withBlck` in `template`, also rename the injected `blck` to `forkyBlck`. - https://github.com/nim-lang/Nim/issues/22698
2023-09-21 10:49:14 +00:00
template aggregate: untyped = forkyBlck.message.body.sync_aggregate
More efficient reward data persistance; Address review comments The new format is based on compressed CSV files in two channels: * Detailed per-epoch data * Aggregated "daily" summaries The use of append-only CSV file speeds up significantly the epoch processing speed during data generation. The use of compression results in smaller storage requirements overall. The use of the aggregated files has a very minor cost in both CPU and storage, but leads to near interactive speed for report generation. Other changes: - Implemented support for graceful shut downs to avoid corrupting the saved files. - Fixed a memory leak caused by lacking `StateCache` clean up on each iteration. - Addressed review comments - Moved the rewards and penalties calculation code in a separate module Required invasive changes to existing modules: - The `data` field of the `KeyedBlockRef` type is made public to be used by the validator rewards monitor's Chain DAG update procedure. - The `getForkedBlock` procedure from the `blockchain_dag.nim` module is made public to be used by the validator rewards monitor's Chain DAG update procedure.
2022-01-17 12:58:33 +00:00
doAssert indices.len == SYNC_COMMITTEE_SIZE
doAssert aggregate.sync_committee_bits.len == SYNC_COMMITTEE_SIZE
switch `withStateAndBlck` usage to `forkyState`; rm unused `tests/mocking/` modules (#4206)
2022-10-03 13:08:50 +00:00
doAssert forkyState.data.current_sync_committee.pubkeys.len ==
SYNC_COMMITTEE_SIZE
More efficient reward data persistance; Address review comments The new format is based on compressed CSV files in two channels: * Detailed per-epoch data * Aggregated "daily" summaries The use of append-only CSV file speeds up significantly the epoch processing speed during data generation. The use of compression results in smaller storage requirements overall. The use of the aggregated files has a very minor cost in both CPU and storage, but leads to near interactive speed for report generation. Other changes: - Implemented support for graceful shut downs to avoid corrupting the saved files. - Fixed a memory leak caused by lacking `StateCache` clean up on each iteration. - Addressed review comments - Moved the rewards and penalties calculation code in a separate module Required invasive changes to existing modules: - The `data` field of the `KeyedBlockRef` type is made public to be used by the validator rewards monitor's Chain DAG update procedure. - The `getForkedBlock` procedure from the `blockchain_dag.nim` module is made public to be used by the validator rewards monitor's Chain DAG update procedure.
2022-01-17 12:58:33 +00:00
for i in 0 ..< SYNC_COMMITTEE_SIZE:
rewardsAndPenalties[indices[i]].max_sync_committee_reward +=
participant_reward
if aggregate.sync_committee_bits[i]:
rewardsAndPenalties[indices[i]].sync_committee_outcome +=
participant_reward.int64
`blck` --> `forkyBlck` when using `withBlck` / `withStateAndBlck` (#5451) For symmetry with `forkyState` when using `withState`, and to avoid problems with shadowing of `blck` when using `withBlck` in `template`, also rename the injected `blck` to `forkyBlck`. - https://github.com/nim-lang/Nim/issues/22698
2023-09-21 10:49:14 +00:00
rewardsAndPenalties[forkyBlck.message.proposer_index]
.proposer_outcome += proposer_reward.int64
More efficient reward data persistance; Address review comments The new format is based on compressed CSV files in two channels: * Detailed per-epoch data * Aggregated "daily" summaries The use of append-only CSV file speeds up significantly the epoch processing speed during data generation. The use of compression results in smaller storage requirements overall. The use of the aggregated files has a very minor cost in both CPU and storage, but leads to near interactive speed for report generation. Other changes: - Implemented support for graceful shut downs to avoid corrupting the saved files. - Fixed a memory leak caused by lacking `StateCache` clean up on each iteration. - Addressed review comments - Moved the rewards and penalties calculation code in a separate module Required invasive changes to existing modules: - The `data` field of the `KeyedBlockRef` type is made public to be used by the validator rewards monitor's Chain DAG update procedure. - The `getForkedBlock` procedure from the `blockchain_dag.nim` module is made public to be used by the validator rewards monitor's Chain DAG update procedure.
2022-01-17 12:58:33 +00:00
else:
rewardsAndPenalties[indices[i]].sync_committee_outcome -=
participant_reward.int64
proc collectBlockRewardsAndPenalties*(
rewardsAndPenalties: var seq[RewardsAndPenalties],
forkedState: ForkedHashedBeaconState,
forkedBlock: ForkedTrustedSignedBeaconBlock,
auxiliaryState: var AuxiliaryState,
cache: var StateCache, cfg: RuntimeConfig) =
rewardsAndPenalties.collectFromProposerSlashings(forkedState, forkedBlock)
rewardsAndPenalties.collectFromAttesterSlashings(forkedState, forkedBlock)
rewardsAndPenalties.collectFromAttestations(
forkedState, forkedBlock, auxiliaryState.epochParticipationFlags, cache)
rewardsAndPenalties.collectFromDeposits(
forkedState, forkedBlock, auxiliaryState.pubkeyToIndex, cfg)
# This table is needed only to resolve double deposits in the same block, so
# it can be cleared after processing all deposits for the current block.
auxiliaryState.pubkeyToIndex.clear
rewardsAndPenalties.collectFromSyncAggregate(forkedState, forkedBlock, cache)
func serializeToCsv*(rp: RewardsAndPenalties,
avgInclusionDelay = none(float)): string =
for name, value in fieldPairs(rp):
if value isnot Option:
result &= $value & ","
if avgInclusionDelay.isSome:
result.addFloat(avgInclusionDelay.get)
elif rp.inclusion_delay.isSome:
result &= $rp.inclusion_delay.get
add EF consensus spec test attester slashing, proposer slashing, voluntary exit, and execution layer withdrawal request tests (#6234)
2024-04-25 08:41:17 +00:00
result &= "\n"