status-im
/
nimbus-eth2
mirror of https://github.com/status-im/nimbus-eth2.git
2
0
Fork 0
Code Issues Projects Releases Wiki Activity
nimbus-eth2/beacon_chain/consensus_object_pools/spec_cache.nim

195 lines
8.2 KiB
Nim
Raw Normal View History

removed the BlockPool type and all of the proxy functions around it (#1401) * removed the BlockPool type and all of the proxy functions around it - passing the chain DAG and the quarantine explicitly where appropriately - they don't need to be bundled in a type * fixed the build after the rebase
2020-07-30 19:18:17 +00:00
# beacon_chain
spec URL & copyright year update (#3338)
2022-01-29 01:05:39 +00:00
# Copyright (c) 2018-2022 Status Research & Development GmbH
removed the BlockPool type and all of the proxy functions around it (#1401) * removed the BlockPool type and all of the proxy functions around it - passing the chain DAG and the quarantine explicitly where appropriately - they don't need to be bundled in a type * fixed the build after the rebase
2020-07-30 19:18:17 +00:00
# Licensed and distributed under either of
# * MIT license (license terms in the root directory or at https://opensource.org/licenses/MIT).
# * Apache v2 license (license terms in the root directory or at https://www.apache.org/licenses/LICENSE-2.0).
# at your option. This file may not be copied, modified, or distributed except according to those terms.
fix broken metrics during replay (#2090) * move metrics out of state transition * add validator count metric * remove expensive beacon_current_validators, beacon_previous_validators metrics (they should be reimplemented with cache), add cheap beacon_active_validators to approximate * remove unused validator count metrics * tidy imports/defects
2020-11-27 22:16:13 +00:00
{.push raises: [Defect].}
removed the BlockPool type and all of the proxy functions around it (#1401) * removed the BlockPool type and all of the proxy functions around it - passing the chain DAG and the quarantine explicitly where appropriately - they don't need to be bundled in a type * fixed the build after the rebase
2020-07-30 19:18:17 +00:00
import
refactor executionPayload tests; reduce HashSet creation (#3003)
2021-10-20 11:36:38 +00:00
std/sequtils,
Harden CommitteeIndex, SubnetId, SyncSubcommitteeIndex (#3259) * Harden CommitteeIndex, SubnetId, SyncSubcommitteeIndex Harden the use of `CommitteeIndex` et al to prevent future issues by using a distinct type, then validating before use in several cases - datatypes in spec are kept simple though so that invalid data still can be read. * fix invalid epoch used in REST `/eth/v1/beacon/states/{state_id}/committees` committee length (could return invalid data) * normalize some variable names * normalize committee index loops * fix `RestAttesterDuty` to use `uint64` for `validator_committee_index` * validate `CommitteeIndex` on ingress in REST API * update rest rules with stricter parsing * better REST serializers * save lots of memory by not using `zip` ...at least a few bytes!
2022-01-08 23:28:49 +00:00
stew/results,
misc memory and perf fixes (#1899) * misc memory and perf fixes * use EpochRef for attestation aggregation * compress effective balances in memory (medalla unfinalized: 4gb -> 1gb) * avoid hitting db when rewinding to head or clearance state * avoid hitting db when blocks can be applied to in-memory state - speeds up startup considerably * avoid storing epochref in fork choice * simplify and speed up beacon block creation flow - avoids state reload thanks to head rewind optimization * iterator-based committee and attestation participation help avoid lots of small memory allocations throughout epoch transition (40% speedup on epoch processing, for example during startup) * add constant for threshold
2020-10-22 10:53:33 +00:00
chronicles,
remove one cache, add another (#1449) * remove one cache, add another This cache removes the need for rewinding in most attestation validation flow since the attestations come from one of two epochs and must be targetting a viable block. Additionally, it also removes all state caches which are less likely to be used over-all - more metrics are needed to track the rewinding. On risk is that when chains don't finalize, we'll have lots of epochrefs in memory meaning lots of validator key databases, most being exactly the same. This can be addressed in any number of ways. Some of the memory usage is mitigated by the fact that we previously had lots of big state caches and now we're keeping only keys instead. * cleanups * doc
2020-08-06 19:48:47 +00:00
../extras,
Harden CommitteeIndex, SubnetId, SyncSubcommitteeIndex (#3259) * Harden CommitteeIndex, SubnetId, SyncSubcommitteeIndex Harden the use of `CommitteeIndex` et al to prevent future issues by using a distinct type, then validating before use in several cases - datatypes in spec are kept simple though so that invalid data still can be read. * fix invalid epoch used in REST `/eth/v1/beacon/states/{state_id}/committees` committee length (could return invalid data) * normalize some variable names * normalize committee index loops * fix `RestAttesterDuty` to use `uint64` for `validator_committee_index` * validate `CommitteeIndex` on ingress in REST API * update rest rules with stricter parsing * better REST serializers * save lots of memory by not using `zip` ...at least a few bytes!
2022-01-08 23:28:49 +00:00
../spec/[beaconstate, helpers, network, signatures, validator],
rework spec imports (#2779) The spec imports are a mess to work with, so this branch cleans them up a bit to ensure that we avoid generic sandwitches and that importing stuff generally becomes easier. * reexport crypto/digest/presets because these are part of the public symbol set of the rest of the spec types * don't export `merge` types from `base` - this causes circular deps * fix circular deps in `ssz/spec_types` - this is the first step in disentangling ssz from spec * be explicit about phase0 vs altair - longer term, `altair` will become the "natural" type set, then merge and so on, so no point in giving `phase0` special preferential treatment
2021-08-12 13:08:20 +00:00
../spec/datatypes/base,
./block_pools_types, blockchain_dag
export
Harden CommitteeIndex, SubnetId, SyncSubcommitteeIndex (#3259) * Harden CommitteeIndex, SubnetId, SyncSubcommitteeIndex Harden the use of `CommitteeIndex` et al to prevent future issues by using a distinct type, then validating before use in several cases - datatypes in spec are kept simple though so that invalid data still can be read. * fix invalid epoch used in REST `/eth/v1/beacon/states/{state_id}/committees` committee length (could return invalid data) * normalize some variable names * normalize committee index loops * fix `RestAttesterDuty` to use `uint64` for `validator_committee_index` * validate `CommitteeIndex` on ingress in REST API * update rest rules with stricter parsing * better REST serializers * save lots of memory by not using `zip` ...at least a few bytes!
2022-01-08 23:28:49 +00:00
base, extras, block_pools_types, results
removed the BlockPool type and all of the proxy functions around it (#1401) * removed the BlockPool type and all of the proxy functions around it - passing the chain DAG and the quarantine explicitly where appropriately - they don't need to be bundled in a type * fixed the build after the rebase
2020-07-30 19:18:17 +00:00
# Spec functions implemented based on cached values instead of the full state
func count_active_validators*(epochInfo: EpochRef): uint64 =
epochInfo.shuffled_active_validator_indices.lenu64
update from phase0/altair v1.1.6 URLs to v1.1.8 spec URLs (#3238)
2022-01-04 03:57:15 +00:00
# https://github.com/ethereum/consensus-specs/blob/v1.1.8/specs/phase0/beacon-chain.md#get_committee_count_per_slot
removed the BlockPool type and all of the proxy functions around it (#1401) * removed the BlockPool type and all of the proxy functions around it - passing the chain DAG and the quarantine explicitly where appropriately - they don't need to be bundled in a type * fixed the build after the rebase
2020-07-30 19:18:17 +00:00
func get_committee_count_per_slot*(epochInfo: EpochRef): uint64 =
get_committee_count_per_slot(count_active_validators(epochInfo))
Perform attestation check before broadcast (#2550) Currently, we have a bit of a convoluted flow where when sending attestations, we start broadcasting them over gossip then pass them to the attestation validation to include them in the local attestation pool - it should be the other way around: we should be checking attestations _before_ gossipping them - this serves as an additional safety net to ensure that we don't publish junk - this becomes more important when publishing attestations from the API. Also, the REST API was performing its own validation meaning attestations coming from REST would be validated twice - finally, the JSON RPC wasn't pre-validating and would happily broadcast invalid attestations. * Unified attestation production pipeline with the same flow for gossip, locally and API-produced attestations: all are now validated and entered into the pool, then broadcast/republished * Refactor subnet handling with specific SubnetId alias, streamlining where subnets are computed, avoiding the need to pass around the number of active validators * Move some of the subnet handling code to eth2_network * Use BitArray throughout for subnet handling
2021-05-10 07:13:36 +00:00
iterator get_committee_indices*(epochRef: EpochRef): CommitteeIndex =
Harden CommitteeIndex, SubnetId, SyncSubcommitteeIndex (#3259) * Harden CommitteeIndex, SubnetId, SyncSubcommitteeIndex Harden the use of `CommitteeIndex` et al to prevent future issues by using a distinct type, then validating before use in several cases - datatypes in spec are kept simple though so that invalid data still can be read. * fix invalid epoch used in REST `/eth/v1/beacon/states/{state_id}/committees` committee length (could return invalid data) * normalize some variable names * normalize committee index loops * fix `RestAttesterDuty` to use `uint64` for `validator_committee_index` * validate `CommitteeIndex` on ingress in REST API * update rest rules with stricter parsing * better REST serializers * save lots of memory by not using `zip` ...at least a few bytes!
2022-01-08 23:28:49 +00:00
let committees_per_slot = get_committee_count_per_slot(epochRef)
for committee_index in get_committee_indices(committees_per_slot):
yield committee_index
func get_committee_index*(epochRef: EpochRef, index: uint64):
Result[CommitteeIndex, cstring] =
check_attestation_index(index, get_committee_count_per_slot(epochRef))
Perform attestation check before broadcast (#2550) Currently, we have a bit of a convoluted flow where when sending attestations, we start broadcasting them over gossip then pass them to the attestation validation to include them in the local attestation pool - it should be the other way around: we should be checking attestations _before_ gossipping them - this serves as an additional safety net to ensure that we don't publish junk - this becomes more important when publishing attestations from the API. Also, the REST API was performing its own validation meaning attestations coming from REST would be validated twice - finally, the JSON RPC wasn't pre-validating and would happily broadcast invalid attestations. * Unified attestation production pipeline with the same flow for gossip, locally and API-produced attestations: all are now validated and entered into the pool, then broadcast/republished * Refactor subnet handling with specific SubnetId alias, streamlining where subnets are computed, avoiding the need to pass around the number of active validators * Move some of the subnet handling code to eth2_network * Use BitArray throughout for subnet handling
2021-05-10 07:13:36 +00:00
update from phase0/altair v1.1.6 URLs to v1.1.8 spec URLs (#3238)
2022-01-04 03:57:15 +00:00
# https://github.com/ethereum/consensus-specs/blob/v1.1.8/specs/phase0/beacon-chain.md#get_beacon_committee
misc memory and perf fixes (#1899) * misc memory and perf fixes * use EpochRef for attestation aggregation * compress effective balances in memory (medalla unfinalized: 4gb -> 1gb) * avoid hitting db when rewinding to head or clearance state * avoid hitting db when blocks can be applied to in-memory state - speeds up startup considerably * avoid storing epochref in fork choice * simplify and speed up beacon block creation flow - avoids state reload thanks to head rewind optimization * iterator-based committee and attestation participation help avoid lots of small memory allocations throughout epoch transition (40% speedup on epoch processing, for example during startup) * add constant for threshold
2020-10-22 10:53:33 +00:00
iterator get_beacon_committee*(
Harden CommitteeIndex, SubnetId, SyncSubcommitteeIndex (#3259) * Harden CommitteeIndex, SubnetId, SyncSubcommitteeIndex Harden the use of `CommitteeIndex` et al to prevent future issues by using a distinct type, then validating before use in several cases - datatypes in spec are kept simple though so that invalid data still can be read. * fix invalid epoch used in REST `/eth/v1/beacon/states/{state_id}/committees` committee length (could return invalid data) * normalize some variable names * normalize committee index loops * fix `RestAttesterDuty` to use `uint64` for `validator_committee_index` * validate `CommitteeIndex` on ingress in REST API * update rest rules with stricter parsing * better REST serializers * save lots of memory by not using `zip` ...at least a few bytes!
2022-01-08 23:28:49 +00:00
epochRef: EpochRef, slot: Slot, committee_index: CommitteeIndex):
(int, ValidatorIndex) =
update 18 spec URLs to v1.1.2; switch from eth2.0-specs to consensus-specs (#2990)
2021-10-14 06:30:21 +00:00
## Return the beacon committee at ``slot`` for ``index``.
misc memory and perf fixes (#1899) * misc memory and perf fixes * use EpochRef for attestation aggregation * compress effective balances in memory (medalla unfinalized: 4gb -> 1gb) * avoid hitting db when rewinding to head or clearance state * avoid hitting db when blocks can be applied to in-memory state - speeds up startup considerably * avoid storing epochref in fork choice * simplify and speed up beacon block creation flow - avoids state reload thanks to head rewind optimization * iterator-based committee and attestation participation help avoid lots of small memory allocations throughout epoch transition (40% speedup on epoch processing, for example during startup) * add constant for threshold
2020-10-22 10:53:33 +00:00
let
committees_per_slot = get_committee_count_per_slot(epochRef)
Harden CommitteeIndex, SubnetId, SyncSubcommitteeIndex (#3259) * Harden CommitteeIndex, SubnetId, SyncSubcommitteeIndex Harden the use of `CommitteeIndex` et al to prevent future issues by using a distinct type, then validating before use in several cases - datatypes in spec are kept simple though so that invalid data still can be read. * fix invalid epoch used in REST `/eth/v1/beacon/states/{state_id}/committees` committee length (could return invalid data) * normalize some variable names * normalize committee index loops * fix `RestAttesterDuty` to use `uint64` for `validator_committee_index` * validate `CommitteeIndex` on ingress in REST API * update rest rules with stricter parsing * better REST serializers * save lots of memory by not using `zip` ...at least a few bytes!
2022-01-08 23:28:49 +00:00
for index_in_committee, idx in compute_committee(
misc memory and perf fixes (#1899) * misc memory and perf fixes * use EpochRef for attestation aggregation * compress effective balances in memory (medalla unfinalized: 4gb -> 1gb) * avoid hitting db when rewinding to head or clearance state * avoid hitting db when blocks can be applied to in-memory state - speeds up startup considerably * avoid storing epochref in fork choice * simplify and speed up beacon block creation flow - avoids state reload thanks to head rewind optimization * iterator-based committee and attestation participation help avoid lots of small memory allocations throughout epoch transition (40% speedup on epoch processing, for example during startup) * add constant for threshold
2020-10-22 10:53:33 +00:00
epochRef.shuffled_active_validator_indices,
Harden CommitteeIndex, SubnetId, SyncSubcommitteeIndex (#3259) * Harden CommitteeIndex, SubnetId, SyncSubcommitteeIndex Harden the use of `CommitteeIndex` et al to prevent future issues by using a distinct type, then validating before use in several cases - datatypes in spec are kept simple though so that invalid data still can be read. * fix invalid epoch used in REST `/eth/v1/beacon/states/{state_id}/committees` committee length (could return invalid data) * normalize some variable names * normalize committee index loops * fix `RestAttesterDuty` to use `uint64` for `validator_committee_index` * validate `CommitteeIndex` on ingress in REST API * update rest rules with stricter parsing * better REST serializers * save lots of memory by not using `zip` ...at least a few bytes!
2022-01-08 23:28:49 +00:00
(slot mod SLOTS_PER_EPOCH) * committees_per_slot + committee_index.asUInt64,
misc memory and perf fixes (#1899) * misc memory and perf fixes * use EpochRef for attestation aggregation * compress effective balances in memory (medalla unfinalized: 4gb -> 1gb) * avoid hitting db when rewinding to head or clearance state * avoid hitting db when blocks can be applied to in-memory state - speeds up startup considerably * avoid storing epochref in fork choice * simplify and speed up beacon block creation flow - avoids state reload thanks to head rewind optimization * iterator-based committee and attestation participation help avoid lots of small memory allocations throughout epoch transition (40% speedup on epoch processing, for example during startup) * add constant for threshold
2020-10-22 10:53:33 +00:00
committees_per_slot * SLOTS_PER_EPOCH
Harden CommitteeIndex, SubnetId, SyncSubcommitteeIndex (#3259) * Harden CommitteeIndex, SubnetId, SyncSubcommitteeIndex Harden the use of `CommitteeIndex` et al to prevent future issues by using a distinct type, then validating before use in several cases - datatypes in spec are kept simple though so that invalid data still can be read. * fix invalid epoch used in REST `/eth/v1/beacon/states/{state_id}/committees` committee length (could return invalid data) * normalize some variable names * normalize committee index loops * fix `RestAttesterDuty` to use `uint64` for `validator_committee_index` * validate `CommitteeIndex` on ingress in REST API * update rest rules with stricter parsing * better REST serializers * save lots of memory by not using `zip` ...at least a few bytes!
2022-01-08 23:28:49 +00:00
): yield (index_in_committee, idx)
misc memory and perf fixes (#1899) * misc memory and perf fixes * use EpochRef for attestation aggregation * compress effective balances in memory (medalla unfinalized: 4gb -> 1gb) * avoid hitting db when rewinding to head or clearance state * avoid hitting db when blocks can be applied to in-memory state - speeds up startup considerably * avoid storing epochref in fork choice * simplify and speed up beacon block creation flow - avoids state reload thanks to head rewind optimization * iterator-based committee and attestation participation help avoid lots of small memory allocations throughout epoch transition (40% speedup on epoch processing, for example during startup) * add constant for threshold
2020-10-22 10:53:33 +00:00
update from phase0/altair v1.1.6 URLs to v1.1.8 spec URLs (#3238)
2022-01-04 03:57:15 +00:00
# https://github.com/ethereum/consensus-specs/blob/v1.1.8/specs/phase0/beacon-chain.md#get_beacon_committee
removed the BlockPool type and all of the proxy functions around it (#1401) * removed the BlockPool type and all of the proxy functions around it - passing the chain DAG and the quarantine explicitly where appropriately - they don't need to be bundled in a type * fixed the build after the rebase
2020-07-30 19:18:17 +00:00
func get_beacon_committee*(
Harden CommitteeIndex, SubnetId, SyncSubcommitteeIndex (#3259) * Harden CommitteeIndex, SubnetId, SyncSubcommitteeIndex Harden the use of `CommitteeIndex` et al to prevent future issues by using a distinct type, then validating before use in several cases - datatypes in spec are kept simple though so that invalid data still can be read. * fix invalid epoch used in REST `/eth/v1/beacon/states/{state_id}/committees` committee length (could return invalid data) * normalize some variable names * normalize committee index loops * fix `RestAttesterDuty` to use `uint64` for `validator_committee_index` * validate `CommitteeIndex` on ingress in REST API * update rest rules with stricter parsing * better REST serializers * save lots of memory by not using `zip` ...at least a few bytes!
2022-01-08 23:28:49 +00:00
epochRef: EpochRef, slot: Slot, committee_index: CommitteeIndex):
seq[ValidatorIndex] =
update 18 spec URLs to v1.1.2; switch from eth2.0-specs to consensus-specs (#2990)
2021-10-14 06:30:21 +00:00
## Return the beacon committee at ``slot`` for ``index``.
removed the BlockPool type and all of the proxy functions around it (#1401) * removed the BlockPool type and all of the proxy functions around it - passing the chain DAG and the quarantine explicitly where appropriately - they don't need to be bundled in a type * fixed the build after the rebase
2020-07-30 19:18:17 +00:00
let
committees_per_slot = get_committee_count_per_slot(epochRef)
compute_committee(
epochRef.shuffled_active_validator_indices,
Harden CommitteeIndex, SubnetId, SyncSubcommitteeIndex (#3259) * Harden CommitteeIndex, SubnetId, SyncSubcommitteeIndex Harden the use of `CommitteeIndex` et al to prevent future issues by using a distinct type, then validating before use in several cases - datatypes in spec are kept simple though so that invalid data still can be read. * fix invalid epoch used in REST `/eth/v1/beacon/states/{state_id}/committees` committee length (could return invalid data) * normalize some variable names * normalize committee index loops * fix `RestAttesterDuty` to use `uint64` for `validator_committee_index` * validate `CommitteeIndex` on ingress in REST API * update rest rules with stricter parsing * better REST serializers * save lots of memory by not using `zip` ...at least a few bytes!
2022-01-08 23:28:49 +00:00
(slot mod SLOTS_PER_EPOCH) * committees_per_slot + committee_index.asUInt64,
removed the BlockPool type and all of the proxy functions around it (#1401) * removed the BlockPool type and all of the proxy functions around it - passing the chain DAG and the quarantine explicitly where appropriately - they don't need to be bundled in a type * fixed the build after the rebase
2020-07-30 19:18:17 +00:00
committees_per_slot * SLOTS_PER_EPOCH
)
update from phase0/altair v1.1.6 URLs to v1.1.8 spec URLs (#3238)
2022-01-04 03:57:15 +00:00
# https://github.com/ethereum/consensus-specs/blob/v1.1.8/specs/phase0/beacon-chain.md#get_beacon_committee
misc memory and perf fixes (#1899) * misc memory and perf fixes * use EpochRef for attestation aggregation * compress effective balances in memory (medalla unfinalized: 4gb -> 1gb) * avoid hitting db when rewinding to head or clearance state * avoid hitting db when blocks can be applied to in-memory state - speeds up startup considerably * avoid storing epochref in fork choice * simplify and speed up beacon block creation flow - avoids state reload thanks to head rewind optimization * iterator-based committee and attestation participation help avoid lots of small memory allocations throughout epoch transition (40% speedup on epoch processing, for example during startup) * add constant for threshold
2020-10-22 10:53:33 +00:00
func get_beacon_committee_len*(
Harden CommitteeIndex, SubnetId, SyncSubcommitteeIndex (#3259) * Harden CommitteeIndex, SubnetId, SyncSubcommitteeIndex Harden the use of `CommitteeIndex` et al to prevent future issues by using a distinct type, then validating before use in several cases - datatypes in spec are kept simple though so that invalid data still can be read. * fix invalid epoch used in REST `/eth/v1/beacon/states/{state_id}/committees` committee length (could return invalid data) * normalize some variable names * normalize committee index loops * fix `RestAttesterDuty` to use `uint64` for `validator_committee_index` * validate `CommitteeIndex` on ingress in REST API * update rest rules with stricter parsing * better REST serializers * save lots of memory by not using `zip` ...at least a few bytes!
2022-01-08 23:28:49 +00:00
epochRef: EpochRef, slot: Slot, committee_index: CommitteeIndex): uint64 =
update 18 spec URLs to v1.1.2; switch from eth2.0-specs to consensus-specs (#2990)
2021-10-14 06:30:21 +00:00
## Return the number of members in the beacon committee at ``slot`` for ``index``.
misc memory and perf fixes (#1899) * misc memory and perf fixes * use EpochRef for attestation aggregation * compress effective balances in memory (medalla unfinalized: 4gb -> 1gb) * avoid hitting db when rewinding to head or clearance state * avoid hitting db when blocks can be applied to in-memory state - speeds up startup considerably * avoid storing epochref in fork choice * simplify and speed up beacon block creation flow - avoids state reload thanks to head rewind optimization * iterator-based committee and attestation participation help avoid lots of small memory allocations throughout epoch transition (40% speedup on epoch processing, for example during startup) * add constant for threshold
2020-10-22 10:53:33 +00:00
let
committees_per_slot = get_committee_count_per_slot(epochRef)
compute_committee_len(
count_active_validators(epochRef),
Harden CommitteeIndex, SubnetId, SyncSubcommitteeIndex (#3259) * Harden CommitteeIndex, SubnetId, SyncSubcommitteeIndex Harden the use of `CommitteeIndex` et al to prevent future issues by using a distinct type, then validating before use in several cases - datatypes in spec are kept simple though so that invalid data still can be read. * fix invalid epoch used in REST `/eth/v1/beacon/states/{state_id}/committees` committee length (could return invalid data) * normalize some variable names * normalize committee index loops * fix `RestAttesterDuty` to use `uint64` for `validator_committee_index` * validate `CommitteeIndex` on ingress in REST API * update rest rules with stricter parsing * better REST serializers * save lots of memory by not using `zip` ...at least a few bytes!
2022-01-08 23:28:49 +00:00
(slot mod SLOTS_PER_EPOCH) * committees_per_slot + committee_index.asUInt64,
misc memory and perf fixes (#1899) * misc memory and perf fixes * use EpochRef for attestation aggregation * compress effective balances in memory (medalla unfinalized: 4gb -> 1gb) * avoid hitting db when rewinding to head or clearance state * avoid hitting db when blocks can be applied to in-memory state - speeds up startup considerably * avoid storing epochref in fork choice * simplify and speed up beacon block creation flow - avoids state reload thanks to head rewind optimization * iterator-based committee and attestation participation help avoid lots of small memory allocations throughout epoch transition (40% speedup on epoch processing, for example during startup) * add constant for threshold
2020-10-22 10:53:33 +00:00
committees_per_slot * SLOTS_PER_EPOCH
)
update from phase0/altair v1.1.6 URLs to v1.1.8 spec URLs (#3238)
2022-01-04 03:57:15 +00:00
# https://github.com/ethereum/consensus-specs/blob/v1.1.8/specs/phase0/beacon-chain.md#get_attesting_indices
misc memory and perf fixes (#1899) * misc memory and perf fixes * use EpochRef for attestation aggregation * compress effective balances in memory (medalla unfinalized: 4gb -> 1gb) * avoid hitting db when rewinding to head or clearance state * avoid hitting db when blocks can be applied to in-memory state - speeds up startup considerably * avoid storing epochref in fork choice * simplify and speed up beacon block creation flow - avoids state reload thanks to head rewind optimization * iterator-based committee and attestation participation help avoid lots of small memory allocations throughout epoch transition (40% speedup on epoch processing, for example during startup) * add constant for threshold
2020-10-22 10:53:33 +00:00
iterator get_attesting_indices*(epochRef: EpochRef,
Harden CommitteeIndex, SubnetId, SyncSubcommitteeIndex (#3259) * Harden CommitteeIndex, SubnetId, SyncSubcommitteeIndex Harden the use of `CommitteeIndex` et al to prevent future issues by using a distinct type, then validating before use in several cases - datatypes in spec are kept simple though so that invalid data still can be read. * fix invalid epoch used in REST `/eth/v1/beacon/states/{state_id}/committees` committee length (could return invalid data) * normalize some variable names * normalize committee index loops * fix `RestAttesterDuty` to use `uint64` for `validator_committee_index` * validate `CommitteeIndex` on ingress in REST API * update rest rules with stricter parsing * better REST serializers * save lots of memory by not using `zip` ...at least a few bytes!
2022-01-08 23:28:49 +00:00
slot: Slot,
committee_index: CommitteeIndex,
misc memory and perf fixes (#1899) * misc memory and perf fixes * use EpochRef for attestation aggregation * compress effective balances in memory (medalla unfinalized: 4gb -> 1gb) * avoid hitting db when rewinding to head or clearance state * avoid hitting db when blocks can be applied to in-memory state - speeds up startup considerably * avoid storing epochref in fork choice * simplify and speed up beacon block creation flow - avoids state reload thanks to head rewind optimization * iterator-based committee and attestation participation help avoid lots of small memory allocations throughout epoch transition (40% speedup on epoch processing, for example during startup) * add constant for threshold
2020-10-22 10:53:33 +00:00
bits: CommitteeValidatorsBits):
ValidatorIndex =
Harden CommitteeIndex, SubnetId, SyncSubcommitteeIndex (#3259) * Harden CommitteeIndex, SubnetId, SyncSubcommitteeIndex Harden the use of `CommitteeIndex` et al to prevent future issues by using a distinct type, then validating before use in several cases - datatypes in spec are kept simple though so that invalid data still can be read. * fix invalid epoch used in REST `/eth/v1/beacon/states/{state_id}/committees` committee length (could return invalid data) * normalize some variable names * normalize committee index loops * fix `RestAttesterDuty` to use `uint64` for `validator_committee_index` * validate `CommitteeIndex` on ingress in REST API * update rest rules with stricter parsing * better REST serializers * save lots of memory by not using `zip` ...at least a few bytes!
2022-01-08 23:28:49 +00:00
if bits.lenu64 != get_beacon_committee_len(epochRef, slot, committee_index):
misc memory and perf fixes (#1899) * misc memory and perf fixes * use EpochRef for attestation aggregation * compress effective balances in memory (medalla unfinalized: 4gb -> 1gb) * avoid hitting db when rewinding to head or clearance state * avoid hitting db when blocks can be applied to in-memory state - speeds up startup considerably * avoid storing epochref in fork choice * simplify and speed up beacon block creation flow - avoids state reload thanks to head rewind optimization * iterator-based committee and attestation participation help avoid lots of small memory allocations throughout epoch transition (40% speedup on epoch processing, for example during startup) * add constant for threshold
2020-10-22 10:53:33 +00:00
trace "get_attesting_indices: inconsistent aggregation and committee length"
else:
Harden CommitteeIndex, SubnetId, SyncSubcommitteeIndex (#3259) * Harden CommitteeIndex, SubnetId, SyncSubcommitteeIndex Harden the use of `CommitteeIndex` et al to prevent future issues by using a distinct type, then validating before use in several cases - datatypes in spec are kept simple though so that invalid data still can be read. * fix invalid epoch used in REST `/eth/v1/beacon/states/{state_id}/committees` committee length (could return invalid data) * normalize some variable names * normalize committee index loops * fix `RestAttesterDuty` to use `uint64` for `validator_committee_index` * validate `CommitteeIndex` on ingress in REST API * update rest rules with stricter parsing * better REST serializers * save lots of memory by not using `zip` ...at least a few bytes!
2022-01-08 23:28:49 +00:00
for index_in_committee, validator_index in get_beacon_committee(
epochRef, slot, committee_index).pairs():
if bits[index_in_committee]:
yield validator_index
misc memory and perf fixes (#1899) * misc memory and perf fixes * use EpochRef for attestation aggregation * compress effective balances in memory (medalla unfinalized: 4gb -> 1gb) * avoid hitting db when rewinding to head or clearance state * avoid hitting db when blocks can be applied to in-memory state - speeds up startup considerably * avoid storing epochref in fork choice * simplify and speed up beacon block creation flow - avoids state reload thanks to head rewind optimization * iterator-based committee and attestation participation help avoid lots of small memory allocations throughout epoch transition (40% speedup on epoch processing, for example during startup) * add constant for threshold
2020-10-22 10:53:33 +00:00
remove attestation/aggregate queue (#2519) With the introduction of batching and lazy attestation aggregation, it no longer makes sense to enqueue attestations between the signature check and adding them to the attestation pool - this only takes up valuable CPU without any real benefit. * add successfully validated attestations to attestion pool directly * avoid copying participant list around for single-vote attestations, pass single validator index instead * release decompressed gossip memory earlier, specially during async message validation * use cooked signatures in a few more places to avoid reloads and errors * remove some Defect-raising versions of signature-loading * release decompressed data memory before validating message
2021-04-26 20:39:44 +00:00
func get_attesting_indices_one*(epochRef: EpochRef,
Harden CommitteeIndex, SubnetId, SyncSubcommitteeIndex (#3259) * Harden CommitteeIndex, SubnetId, SyncSubcommitteeIndex Harden the use of `CommitteeIndex` et al to prevent future issues by using a distinct type, then validating before use in several cases - datatypes in spec are kept simple though so that invalid data still can be read. * fix invalid epoch used in REST `/eth/v1/beacon/states/{state_id}/committees` committee length (could return invalid data) * normalize some variable names * normalize committee index loops * fix `RestAttesterDuty` to use `uint64` for `validator_committee_index` * validate `CommitteeIndex` on ingress in REST API * update rest rules with stricter parsing * better REST serializers * save lots of memory by not using `zip` ...at least a few bytes!
2022-01-08 23:28:49 +00:00
slot: Slot,
committee_index: CommitteeIndex,
remove attestation/aggregate queue (#2519) With the introduction of batching and lazy attestation aggregation, it no longer makes sense to enqueue attestations between the signature check and adding them to the attestation pool - this only takes up valuable CPU without any real benefit. * add successfully validated attestations to attestion pool directly * avoid copying participant list around for single-vote attestations, pass single validator index instead * release decompressed gossip memory earlier, specially during async message validation * use cooked signatures in a few more places to avoid reloads and errors * remove some Defect-raising versions of signature-loading * release decompressed data memory before validating message
2021-04-26 20:39:44 +00:00
bits: CommitteeValidatorsBits):
Option[ValidatorIndex] =
# A variation on get_attesting_indices that returns the validator index only
# if only one validator index is set
Harden CommitteeIndex, SubnetId, SyncSubcommitteeIndex (#3259) * Harden CommitteeIndex, SubnetId, SyncSubcommitteeIndex Harden the use of `CommitteeIndex` et al to prevent future issues by using a distinct type, then validating before use in several cases - datatypes in spec are kept simple though so that invalid data still can be read. * fix invalid epoch used in REST `/eth/v1/beacon/states/{state_id}/committees` committee length (could return invalid data) * normalize some variable names * normalize committee index loops * fix `RestAttesterDuty` to use `uint64` for `validator_committee_index` * validate `CommitteeIndex` on ingress in REST API * update rest rules with stricter parsing * better REST serializers * save lots of memory by not using `zip` ...at least a few bytes!
2022-01-08 23:28:49 +00:00
var res = none(ValidatorIndex)
for validator_index in get_attesting_indices(
epochRef, slot, committee_index, bits):
if res.isSome(): return none(ValidatorIndex)
res = some(validator_index)
res
remove attestation/aggregate queue (#2519) With the introduction of batching and lazy attestation aggregation, it no longer makes sense to enqueue attestations between the signature check and adding them to the attestation pool - this only takes up valuable CPU without any real benefit. * add successfully validated attestations to attestion pool directly * avoid copying participant list around for single-vote attestations, pass single validator index instead * release decompressed gossip memory earlier, specially during async message validation * use cooked signatures in a few more places to avoid reloads and errors * remove some Defect-raising versions of signature-loading * release decompressed data memory before validating message
2021-04-26 20:39:44 +00:00
update from phase0/altair v1.1.6 URLs to v1.1.8 spec URLs (#3238)
2022-01-04 03:57:15 +00:00
# https://github.com/ethereum/consensus-specs/blob/v1.1.8/specs/phase0/beacon-chain.md#get_attesting_indices
removed the BlockPool type and all of the proxy functions around it (#1401) * removed the BlockPool type and all of the proxy functions around it - passing the chain DAG and the quarantine explicitly where appropriately - they don't need to be bundled in a type * fixed the build after the rebase
2020-07-30 19:18:17 +00:00
func get_attesting_indices*(epochRef: EpochRef,
Harden CommitteeIndex, SubnetId, SyncSubcommitteeIndex (#3259) * Harden CommitteeIndex, SubnetId, SyncSubcommitteeIndex Harden the use of `CommitteeIndex` et al to prevent future issues by using a distinct type, then validating before use in several cases - datatypes in spec are kept simple though so that invalid data still can be read. * fix invalid epoch used in REST `/eth/v1/beacon/states/{state_id}/committees` committee length (could return invalid data) * normalize some variable names * normalize committee index loops * fix `RestAttesterDuty` to use `uint64` for `validator_committee_index` * validate `CommitteeIndex` on ingress in REST API * update rest rules with stricter parsing * better REST serializers * save lots of memory by not using `zip` ...at least a few bytes!
2022-01-08 23:28:49 +00:00
slot: Slot,
committee_index: CommitteeIndex,
removed the BlockPool type and all of the proxy functions around it (#1401) * removed the BlockPool type and all of the proxy functions around it - passing the chain DAG and the quarantine explicitly where appropriately - they don't need to be bundled in a type * fixed the build after the rebase
2020-07-30 19:18:17 +00:00
bits: CommitteeValidatorsBits):
remove several IntSet usages in lieu of seq[ValidatorIndex] (#2288) * remove several IntSet usages in lieu of seq[ValidatorIndex] * convert smaller types to larger types * larger type, again
2021-02-08 07:27:30 +00:00
seq[ValidatorIndex] =
# TODO sequtils2 mapIt
Harden CommitteeIndex, SubnetId, SyncSubcommitteeIndex (#3259) * Harden CommitteeIndex, SubnetId, SyncSubcommitteeIndex Harden the use of `CommitteeIndex` et al to prevent future issues by using a distinct type, then validating before use in several cases - datatypes in spec are kept simple though so that invalid data still can be read. * fix invalid epoch used in REST `/eth/v1/beacon/states/{state_id}/committees` committee length (could return invalid data) * normalize some variable names * normalize committee index loops * fix `RestAttesterDuty` to use `uint64` for `validator_committee_index` * validate `CommitteeIndex` on ingress in REST API * update rest rules with stricter parsing * better REST serializers * save lots of memory by not using `zip` ...at least a few bytes!
2022-01-08 23:28:49 +00:00
for idx in get_attesting_indices(epochRef, slot, committee_index, bits):
remove several IntSet usages in lieu of seq[ValidatorIndex] (#2288) * remove several IntSet usages in lieu of seq[ValidatorIndex] * convert smaller types to larger types * larger type, again
2021-02-08 07:27:30 +00:00
result.add(idx)
removed the BlockPool type and all of the proxy functions around it (#1401) * removed the BlockPool type and all of the proxy functions around it - passing the chain DAG and the quarantine explicitly where appropriately - they don't need to be bundled in a type * fixed the build after the rebase
2020-07-30 19:18:17 +00:00
update from phase0/altair v1.1.6 URLs to v1.1.8 spec URLs (#3238)
2022-01-04 03:57:15 +00:00
# https://github.com/ethereum/consensus-specs/blob/v1.1.8/specs/phase0/beacon-chain.md#is_valid_indexed_attestation
implement clock disparity for attestation validation (#1568) This implements disparity, resolving a part of https://github.com/status-im/nim-beacon-chain/issues/1367 * make BeaconTime a duration for fractional seconds * factor out attestation/aggregate validation * simplify recording of queued attestations * simplify attestation signature check * fix blocks_received metric * add some trivial validation tests * remove unresolved attestation table - attestations for unknown blocks are dropped instead (cannot verify their signature)
2020-08-27 07:34:12 +00:00
proc is_valid_indexed_attestation*(
fork: Fork, genesis_validators_root: Eth2Digest,
attestation processing speedups * avoid creating indexed attestation just to check signatures - above all, don't create it when not checking signatures ;) * avoid pointer op when adding attestation to pool * better iterator for yielding attestations * add metric / log for attestation packing time
2021-04-14 14:43:29 +00:00
epochRef: EpochRef,
implement clock disparity for attestation validation (#1568) This implements disparity, resolving a part of https://github.com/status-im/nim-beacon-chain/issues/1367 * make BeaconTime a duration for fractional seconds * factor out attestation/aggregate validation * simplify recording of queued attestations * simplify attestation signature check * fix blocks_received metric * add some trivial validation tests * remove unresolved attestation table - attestations for unknown blocks are dropped instead (cannot verify their signature)
2020-08-27 07:34:12 +00:00
attestation: SomeAttestation, flags: UpdateFlags): Result[void, cstring] =
# This is a variation on `is_valid_indexed_attestation` that works directly
# with an attestation instead of first constructing an `IndexedAttestation`
# and then validating it - for the purpose of validating the signature, the
# order doesn't matter and we can proceed straight to validating the
# signature instead
attestation processing speedups * avoid creating indexed attestation just to check signatures - above all, don't create it when not checking signatures ;) * avoid pointer op when adding attestation to pool * better iterator for yielding attestations * add metric / log for attestation packing time
2021-04-14 14:43:29 +00:00
let sigs = attestation.aggregation_bits.countOnes()
if sigs == 0:
return err("is_valid_indexed_attestation: no attesting indices")
implement clock disparity for attestation validation (#1568) This implements disparity, resolving a part of https://github.com/status-im/nim-beacon-chain/issues/1367 * make BeaconTime a duration for fractional seconds * factor out attestation/aggregate validation * simplify recording of queued attestations * simplify attestation signature check * fix blocks_received metric * add some trivial validation tests * remove unresolved attestation table - attestations for unknown blocks are dropped instead (cannot verify their signature)
2020-08-27 07:34:12 +00:00
# Verify aggregate signature
avoid copying keys for trusted signatures this removes `is_valid_indexed_attestion` from performance benchmarks when replaying blocks
2021-01-25 16:09:31 +00:00
if not (skipBLSValidation in flags or attestation.signature is TrustedSig):
attestation processing speedups * avoid creating indexed attestation just to check signatures - above all, don't create it when not checking signatures ;) * avoid pointer op when adding attestation to pool * better iterator for yielding attestations * add metric / log for attestation packing time
2021-04-14 14:43:29 +00:00
var
singe validator key cache Instead of keeping a validator key list per EpochRef, this PR introduces a single shared validator key list in ChainDAG, and cleans up some other ChainDAG and key-related issues. The PR does not introduce the validator key list in the state transition - this is because we batch-check all signatures before entering the spec code, thus the spec code never hits the cache. A future refactor should _probably_ remove the threadvar altogether. There's a few other small fixes in here that make the flow easier to read: * fix `var ChainDAGRef` -> `ChainDAGRef` * fix `var QuarantineRef` -> `QuarantineRef` * consistent `dag` variable name * avoid using threadvar pubkey cache in most cases * better error messages in batch signature checking
2021-06-01 11:13:40 +00:00
pubkeys = newSeqOfCap[CookedPubKey](sigs)
attestation processing speedups * avoid creating indexed attestation just to check signatures - above all, don't create it when not checking signatures ;) * avoid pointer op when adding attestation to pool * better iterator for yielding attestations * add metric / log for attestation packing time
2021-04-14 14:43:29 +00:00
for index in get_attesting_indices(
epochRef, attestation.data, attestation.aggregation_bits):
write uncompressed validator keys to database (#2639) * write uncompressed validator keys to database Loading 150k+ validator keys on startup in compressed format takes a lot of time - better store them in uncompressed format which makes behaviour just after startup faster / more predictable. * refactor cached validator key access * fix isomorphic cast to work with non-var instances * remove cooked pubkey cache - directly use database cache in chaindag as well (one less cache to keep in sync) * bump blscurve, introduce loadValid for known-to-be-valid keys
2021-06-10 07:37:02 +00:00
pubkeys.add(epochRef.validatorKey(index).get())
attestation processing speedups * avoid creating indexed attestation just to check signatures - above all, don't create it when not checking signatures ;) * avoid pointer op when adding attestation to pool * better iterator for yielding attestations * add metric / log for attestation packing time
2021-04-14 14:43:29 +00:00
implement clock disparity for attestation validation (#1568) This implements disparity, resolving a part of https://github.com/status-im/nim-beacon-chain/issues/1367 * make BeaconTime a duration for fractional seconds * factor out attestation/aggregate validation * simplify recording of queued attestations * simplify attestation signature check * fix blocks_received metric * add some trivial validation tests * remove unresolved attestation table - attestations for unknown blocks are dropped instead (cannot verify their signature)
2020-08-27 07:34:12 +00:00
if not verify_attestation_signature(
fork, genesis_validators_root, attestation.data,
pubkeys, attestation.signature):
attestation processing speedups * avoid creating indexed attestation just to check signatures - above all, don't create it when not checking signatures ;) * avoid pointer op when adding attestation to pool * better iterator for yielding attestations * add metric / log for attestation packing time
2021-04-14 14:43:29 +00:00
return err("is_valid_indexed_attestation: signature verification failure")
remove one cache, add another (#1449) * remove one cache, add another This cache removes the need for rewinding in most attestation validation flow since the attestations come from one of two epochs and must be targetting a viable block. Additionally, it also removes all state caches which are less likely to be used over-all - more metrics are needed to track the rewinding. On risk is that when chains don't finalize, we'll have lots of epochrefs in memory meaning lots of validator key databases, most being exactly the same. This can be addressed in any number of ways. Some of the memory usage is mitigated by the fact that we previously had lots of big state caches and now we're keeping only keys instead. * cleanups * doc
2020-08-06 19:48:47 +00:00
small spec cleanups (#1501) * clean up logging a bit * return error on indexed attestation check
2020-08-13 13:47:06 +00:00
ok()
use epochcache for attesting (#1478)
2020-08-10 13:21:31 +00:00
func makeAttestationData*(
epochRef: EpochRef, bs: BlockSlot,
Datatype cleanup (#1953) * clear up spec todo * test fix * remove unnecessary toSszType * type * one more
2020-11-04 21:52:47 +00:00
committee_index: CommitteeIndex): AttestationData =
use epochcache for attesting (#1478)
2020-08-10 13:21:31 +00:00
## Create an attestation / vote for the block `bs` using the
## data in `epochRef` to fill in the rest of the fields.
## `epochRef` is the epoch information corresponding to the `bs` advanced to
## the slot we're attesting to.
let
slot = bs.slot
time: spring cleaning (#3262) Time in the beacon chain is expressed relative to the genesis time - this PR creates a `beacon_time` module that collects helpers and utilities for dealing the time units - the new module does not deal with actual wall time (that's remains in `beacon_clock`). Collecting the time related stuff in one place makes it easier to find, avoids some circular imports and allows more easily identifying the code actually needs wall time to operate. * move genesis-time-related functionality into `spec/beacon_time` * avoid using `chronos.Duration` for time differences - it does not support negative values (such as when something happens earlier than it should) * saturate conversions between `FAR_FUTURE_XXX`, so as to avoid overflows * fix delay reporting in validator client so it uses the expected deadline of the slot, not "closest wall slot" * simplify looping over the slots of an epoch * `compute_start_slot_at_epoch` -> `start_slot` * `compute_epoch_at_slot` -> `epoch` A follow-up PR will (likely) introduce saturating arithmetic for the time units - this is merely code moves, renames and fixing of small bugs.
2022-01-11 10:01:54 +00:00
current_epoch = slot.epoch()
epoch_boundary_slot = current_epoch.start_slot()
use epochcache for attesting (#1478)
2020-08-10 13:21:31 +00:00
epoch_boundary_block = bs.blck.atSlot(epoch_boundary_slot)
doAssert current_epoch == epochRef.epoch
spec URL & copyright year update (#3338)
2022-01-29 01:05:39 +00:00
# https://github.com/ethereum/consensus-specs/blob/v1.1.9/specs/phase0/validator.md#attestation-data
use epochcache for attesting (#1478)
2020-08-10 13:21:31 +00:00
AttestationData(
slot: slot,
Harden CommitteeIndex, SubnetId, SyncSubcommitteeIndex (#3259) * Harden CommitteeIndex, SubnetId, SyncSubcommitteeIndex Harden the use of `CommitteeIndex` et al to prevent future issues by using a distinct type, then validating before use in several cases - datatypes in spec are kept simple though so that invalid data still can be read. * fix invalid epoch used in REST `/eth/v1/beacon/states/{state_id}/committees` committee length (could return invalid data) * normalize some variable names * normalize committee index loops * fix `RestAttesterDuty` to use `uint64` for `validator_committee_index` * validate `CommitteeIndex` on ingress in REST API * update rest rules with stricter parsing * better REST serializers * save lots of memory by not using `zip` ...at least a few bytes!
2022-01-08 23:28:49 +00:00
index: committee_index.asUInt64,
use epochcache for attesting (#1478)
2020-08-10 13:21:31 +00:00
beacon_block_root: bs.blck.root,
source: epochRef.current_justified_checkpoint,
target: Checkpoint(
epoch: current_epoch,
root: epoch_boundary_block.blck.root
)
)
use epochref for get_committee_assignments() to avoid repeated shuffling (#2463) * use epochref for get_committee_assignments() to avoid repeated shuffling * remove unnecessary imports and StateCache() construction
2021-03-30 15:01:47 +00:00
spec URL & copyright year update (#3338)
2022-01-29 01:05:39 +00:00
# https://github.com/ethereum/consensus-specs/blob/v1.1.9/specs/phase0/validator.md#validator-assignments
use epochref for get_committee_assignments() to avoid repeated shuffling (#2463) * use epochref for get_committee_assignments() to avoid repeated shuffling * remove unnecessary imports and StateCache() construction
2021-03-30 15:01:47 +00:00
iterator get_committee_assignments*(
refactor executionPayload tests; reduce HashSet creation (#3003)
2021-10-20 11:36:38 +00:00
epochRef: EpochRef, validator_indices: HashSet[ValidatorIndex]):
Harden CommitteeIndex, SubnetId, SyncSubcommitteeIndex (#3259) * Harden CommitteeIndex, SubnetId, SyncSubcommitteeIndex Harden the use of `CommitteeIndex` et al to prevent future issues by using a distinct type, then validating before use in several cases - datatypes in spec are kept simple though so that invalid data still can be read. * fix invalid epoch used in REST `/eth/v1/beacon/states/{state_id}/committees` committee length (could return invalid data) * normalize some variable names * normalize committee index loops * fix `RestAttesterDuty` to use `uint64` for `validator_committee_index` * validate `CommitteeIndex` on ingress in REST API * update rest rules with stricter parsing * better REST serializers * save lots of memory by not using `zip` ...at least a few bytes!
2022-01-08 23:28:49 +00:00
tuple[committee_index: CommitteeIndex,
Perform attestation check before broadcast (#2550) Currently, we have a bit of a convoluted flow where when sending attestations, we start broadcasting them over gossip then pass them to the attestation validation to include them in the local attestation pool - it should be the other way around: we should be checking attestations _before_ gossipping them - this serves as an additional safety net to ensure that we don't publish junk - this becomes more important when publishing attestations from the API. Also, the REST API was performing its own validation meaning attestations coming from REST would be validated twice - finally, the JSON RPC wasn't pre-validating and would happily broadcast invalid attestations. * Unified attestation production pipeline with the same flow for gossip, locally and API-produced attestations: all are now validated and entered into the pool, then broadcast/republished * Refactor subnet handling with specific SubnetId alias, streamlining where subnets are computed, avoiding the need to pass around the number of active validators * Move some of the subnet handling code to eth2_network * Use BitArray throughout for subnet handling
2021-05-10 07:13:36 +00:00
subnet_id: SubnetId, slot: Slot] =
use epochref for get_committee_assignments() to avoid repeated shuffling (#2463) * use epochref for get_committee_assignments() to avoid repeated shuffling * remove unnecessary imports and StateCache() construction
2021-03-30 15:01:47 +00:00
let
committees_per_slot = get_committee_count_per_slot(epochRef)
register vc duties with subnet tracker (#2949) * register vc duties with subnet tracker * fix activation logging during startup * cache slot signature to avoid duplicate signature work * schedule aggregation duties one slot at a time to avoid CPU spike at each epoch * lower aggregation subnet pre-subscription time to 4 slots (lowers bandwidth and CPU usage) * update stability subnets in ENR on startup * log gossip state * perform gossip subscriptions just before the next slot starts * document stuff * add random include * don't overwrite subscription state when not subscribed * log target gossip state * updating gossip status once is enough * add test * remove syncQueueLen - this one is not updated at the end of the sync and may cause gossip to disconnect itself completely - use a simple head distance instead * fix gossip disconnection - if in hysteresis, node.gossipState will be set to disabled even though we don't disable topic subscriptions * fix extra duty registration call
2021-10-18 09:11:44 +00:00
epoch = epochRef.epoch
use epochref for get_committee_assignments() to avoid repeated shuffling (#2463) * use epochref for get_committee_assignments() to avoid repeated shuffling * remove unnecessary imports and StateCache() construction
2021-03-30 15:01:47 +00:00
time: spring cleaning (#3262) Time in the beacon chain is expressed relative to the genesis time - this PR creates a `beacon_time` module that collects helpers and utilities for dealing the time units - the new module does not deal with actual wall time (that's remains in `beacon_clock`). Collecting the time related stuff in one place makes it easier to find, avoids some circular imports and allows more easily identifying the code actually needs wall time to operate. * move genesis-time-related functionality into `spec/beacon_time` * avoid using `chronos.Duration` for time differences - it does not support negative values (such as when something happens earlier than it should) * saturate conversions between `FAR_FUTURE_XXX`, so as to avoid overflows * fix delay reporting in validator client so it uses the expected deadline of the slot, not "closest wall slot" * simplify looping over the slots of an epoch * `compute_start_slot_at_epoch` -> `start_slot` * `compute_epoch_at_slot` -> `epoch` A follow-up PR will (likely) introduce saturating arithmetic for the time units - this is merely code moves, renames and fixing of small bugs.
2022-01-11 10:01:54 +00:00
for slot in epoch.slots():
Harden CommitteeIndex, SubnetId, SyncSubcommitteeIndex (#3259) * Harden CommitteeIndex, SubnetId, SyncSubcommitteeIndex Harden the use of `CommitteeIndex` et al to prevent future issues by using a distinct type, then validating before use in several cases - datatypes in spec are kept simple though so that invalid data still can be read. * fix invalid epoch used in REST `/eth/v1/beacon/states/{state_id}/committees` committee length (could return invalid data) * normalize some variable names * normalize committee index loops * fix `RestAttesterDuty` to use `uint64` for `validator_committee_index` * validate `CommitteeIndex` on ingress in REST API * update rest rules with stricter parsing * better REST serializers * save lots of memory by not using `zip` ...at least a few bytes!
2022-01-08 23:28:49 +00:00
for committee_index in get_committee_indices(committees_per_slot):
if anyIt(get_beacon_committee(epochRef, slot, committee_index), it in validator_indices):
use epochref for get_committee_assignments() to avoid repeated shuffling (#2463) * use epochref for get_committee_assignments() to avoid repeated shuffling * remove unnecessary imports and StateCache() construction
2021-03-30 15:01:47 +00:00
yield (
Harden CommitteeIndex, SubnetId, SyncSubcommitteeIndex (#3259) * Harden CommitteeIndex, SubnetId, SyncSubcommitteeIndex Harden the use of `CommitteeIndex` et al to prevent future issues by using a distinct type, then validating before use in several cases - datatypes in spec are kept simple though so that invalid data still can be read. * fix invalid epoch used in REST `/eth/v1/beacon/states/{state_id}/committees` committee length (could return invalid data) * normalize some variable names * normalize committee index loops * fix `RestAttesterDuty` to use `uint64` for `validator_committee_index` * validate `CommitteeIndex` on ingress in REST API * update rest rules with stricter parsing * better REST serializers * save lots of memory by not using `zip` ...at least a few bytes!
2022-01-08 23:28:49 +00:00
committee_index,
compute_subnet_for_attestation(committees_per_slot, slot, committee_index),
use epochref for get_committee_assignments() to avoid repeated shuffling (#2463) * use epochref for get_committee_assignments() to avoid repeated shuffling * remove unnecessary imports and StateCache() construction
2021-03-30 15:01:47 +00:00
slot)
cleanups (#2809) * cleanups * use ForkedTrustedSignedBeaconBlock.ionit where appropriate * move `is_aggregator` to `spec/` * use `errReject` in a few more places * update enr fork id when time is auspicious * use network broadcast functions * Return Ignore for aggregate signature validation timeouts ...consistently between aggregates and attestations. * clean up some more reject/ignore rules * shorten texts a bit * errReject->checkedReject, use err helpers throughout * get rid of quarantine in exitpool as well
2021-08-24 19:49:51 +00:00
func is_aggregator*(epochRef: EpochRef, slot: Slot, index: CommitteeIndex,
slot_signature: ValidatorSig): bool =
let
committee_len = get_beacon_committee_len(epochRef, slot, index)
return is_aggregator(committee_len, slot_signature)