status-im
/
nimbus-eth2
mirror of https://github.com/status-im/nimbus-eth2.git
2
0
Fork 0
Code Issues Projects Releases Wiki Activity
nimbus-eth2/beacon_chain/consensus_object_pools/spec_cache.nim

249 lines
10 KiB
Nim
Raw Normal View History

removed the BlockPool type and all of the proxy functions around it (#1401) * removed the BlockPool type and all of the proxy functions around it - passing the chain DAG and the quarantine explicitly where appropriately - they don't need to be bundled in a type * fixed the build after the rebase
2020-07-30 19:18:17 +00:00
# beacon_chain
use IntSet rather than HashSet[ValidatorIndex] (#2267) * use IntSet rather than HashSet[ValidatorIndex] * add bounds check before uint64 -> int conversion * use intsets in block transitions * remove superfluous Nim issue explanation/reference
2021-01-26 11:52:00 +00:00
# Copyright (c) 2018-2021 Status Research & Development GmbH
removed the BlockPool type and all of the proxy functions around it (#1401) * removed the BlockPool type and all of the proxy functions around it - passing the chain DAG and the quarantine explicitly where appropriately - they don't need to be bundled in a type * fixed the build after the rebase
2020-07-30 19:18:17 +00:00
# Licensed and distributed under either of
# * MIT license (license terms in the root directory or at https://opensource.org/licenses/MIT).
# * Apache v2 license (license terms in the root directory or at https://www.apache.org/licenses/LICENSE-2.0).
# at your option. This file may not be copied, modified, or distributed except according to those terms.
fix broken metrics during replay (#2090) * move metrics out of state transition * add validator count metric * remove expensive beacon_current_validators, beacon_previous_validators metrics (they should be reimplemented with cache), add cheap beacon_active_validators to approximate * remove unused validator count metrics * tidy imports/defects
2020-11-27 22:16:13 +00:00
{.push raises: [Defect].}
removed the BlockPool type and all of the proxy functions around it (#1401) * removed the BlockPool type and all of the proxy functions around it - passing the chain DAG and the quarantine explicitly where appropriately - they don't need to be bundled in a type * fixed the build after the rebase
2020-07-30 19:18:17 +00:00
import
use epochref for get_committee_assignments() to avoid repeated shuffling (#2463) * use epochref for get_committee_assignments() to avoid repeated shuffling * remove unnecessary imports and StateCache() construction
2021-03-30 15:01:47 +00:00
std/[algorithm, intsets, sequtils],
misc memory and perf fixes (#1899) * misc memory and perf fixes * use EpochRef for attestation aggregation * compress effective balances in memory (medalla unfinalized: 4gb -> 1gb) * avoid hitting db when rewinding to head or clearance state * avoid hitting db when blocks can be applied to in-memory state - speeds up startup considerably * avoid storing epochref in fork choice * simplify and speed up beacon block creation flow - avoids state reload thanks to head rewind optimization * iterator-based committee and attestation participation help avoid lots of small memory allocations throughout epoch transition (40% speedup on epoch processing, for example during startup) * add constant for threshold
2020-10-22 10:53:33 +00:00
chronicles,
remove one cache, add another (#1449) * remove one cache, add another This cache removes the need for rewinding in most attestation validation flow since the attestations come from one of two epochs and must be targetting a viable block. Additionally, it also removes all state caches which are less likely to be used over-all - more metrics are needed to track the rewinding. On risk is that when chains don't finalize, we'll have lots of epochrefs in memory meaning lots of validator key databases, most being exactly the same. This can be addressed in any number of ways. Some of the memory usage is mitigated by the fact that we previously had lots of big state caches and now we're keeping only keys instead. * cleanups * doc
2020-08-06 19:48:47 +00:00
../spec/[
use epochref for get_committee_assignments() to avoid repeated shuffling (#2463) * use epochref for get_committee_assignments() to avoid repeated shuffling * remove unnecessary imports and StateCache() construction
2021-03-30 15:01:47 +00:00
crypto, datatypes, digest, helpers, network, presets, signatures,
remove one cache, add another (#1449) * remove one cache, add another This cache removes the need for rewinding in most attestation validation flow since the attestations come from one of two epochs and must be targetting a viable block. Additionally, it also removes all state caches which are less likely to be used over-all - more metrics are needed to track the rewinding. On risk is that when chains don't finalize, we'll have lots of epochrefs in memory meaning lots of validator key databases, most being exactly the same. This can be addressed in any number of ways. Some of the memory usage is mitigated by the fact that we previously had lots of big state caches and now we're keeping only keys instead. * cleanups * doc
2020-08-06 19:48:47 +00:00
validator],
../extras,
Consensus object pools [reorg 4/5] (#2374) * Add documentation * make test doesn't try to build the beacon node :/
2021-03-04 09:13:44 +00:00
./block_pools_types, ./blockchain_dag
removed the BlockPool type and all of the proxy functions around it (#1401) * removed the BlockPool type and all of the proxy functions around it - passing the chain DAG and the quarantine explicitly where appropriately - they don't need to be bundled in a type * fixed the build after the rebase
2020-07-30 19:18:17 +00:00
# Spec functions implemented based on cached values instead of the full state
func count_active_validators*(epochInfo: EpochRef): uint64 =
epochInfo.shuffled_active_validator_indices.lenu64
bump spec refs from v1.0.0 to v1.0.1 and update copyright years (#2357)
2021-02-25 13:37:22 +00:00
# https://github.com/ethereum/eth2.0-specs/blob/v1.0.1/specs/phase0/beacon-chain.md#get_committee_count_per_slot
removed the BlockPool type and all of the proxy functions around it (#1401) * removed the BlockPool type and all of the proxy functions around it - passing the chain DAG and the quarantine explicitly where appropriately - they don't need to be bundled in a type * fixed the build after the rebase
2020-07-30 19:18:17 +00:00
func get_committee_count_per_slot*(epochInfo: EpochRef): uint64 =
get_committee_count_per_slot(count_active_validators(epochInfo))
Perform attestation check before broadcast (#2550) Currently, we have a bit of a convoluted flow where when sending attestations, we start broadcasting them over gossip then pass them to the attestation validation to include them in the local attestation pool - it should be the other way around: we should be checking attestations _before_ gossipping them - this serves as an additional safety net to ensure that we don't publish junk - this becomes more important when publishing attestations from the API. Also, the REST API was performing its own validation meaning attestations coming from REST would be validated twice - finally, the JSON RPC wasn't pre-validating and would happily broadcast invalid attestations. * Unified attestation production pipeline with the same flow for gossip, locally and API-produced attestations: all are now validated and entered into the pool, then broadcast/republished * Refactor subnet handling with specific SubnetId alias, streamlining where subnets are computed, avoiding the need to pass around the number of active validators * Move some of the subnet handling code to eth2_network * Use BitArray throughout for subnet handling
2021-05-10 07:13:36 +00:00
iterator get_committee_indices*(epochRef: EpochRef): CommitteeIndex =
for i in 0'u64..<get_committee_count_per_slot(epochRef):
yield CommitteeIndex(i)
bump spec refs from v1.0.0 to v1.0.1 and update copyright years (#2357)
2021-02-25 13:37:22 +00:00
# https://github.com/ethereum/eth2.0-specs/blob/v1.0.1/specs/phase0/beacon-chain.md#get_beacon_committee
misc memory and perf fixes (#1899) * misc memory and perf fixes * use EpochRef for attestation aggregation * compress effective balances in memory (medalla unfinalized: 4gb -> 1gb) * avoid hitting db when rewinding to head or clearance state * avoid hitting db when blocks can be applied to in-memory state - speeds up startup considerably * avoid storing epochref in fork choice * simplify and speed up beacon block creation flow - avoids state reload thanks to head rewind optimization * iterator-based committee and attestation participation help avoid lots of small memory allocations throughout epoch transition (40% speedup on epoch processing, for example during startup) * add constant for threshold
2020-10-22 10:53:33 +00:00
iterator get_beacon_committee*(
epochRef: EpochRef, slot: Slot, index: CommitteeIndex): ValidatorIndex =
# Return the beacon committee at ``slot`` for ``index``.
let
committees_per_slot = get_committee_count_per_slot(epochRef)
for idx in compute_committee(
epochRef.shuffled_active_validator_indices,
(slot mod SLOTS_PER_EPOCH) * committees_per_slot +
index.uint64,
committees_per_slot * SLOTS_PER_EPOCH
): yield idx
bump spec refs from v1.0.0 to v1.0.1 and update copyright years (#2357)
2021-02-25 13:37:22 +00:00
# https://github.com/ethereum/eth2.0-specs/blob/v1.0.1/specs/phase0/beacon-chain.md#get_beacon_committee
removed the BlockPool type and all of the proxy functions around it (#1401) * removed the BlockPool type and all of the proxy functions around it - passing the chain DAG and the quarantine explicitly where appropriately - they don't need to be bundled in a type * fixed the build after the rebase
2020-07-30 19:18:17 +00:00
func get_beacon_committee*(
epochRef: EpochRef, slot: Slot, index: CommitteeIndex): seq[ValidatorIndex] =
# Return the beacon committee at ``slot`` for ``index``.
let
committees_per_slot = get_committee_count_per_slot(epochRef)
compute_committee(
epochRef.shuffled_active_validator_indices,
(slot mod SLOTS_PER_EPOCH) * committees_per_slot +
index.uint64,
committees_per_slot * SLOTS_PER_EPOCH
)
bump spec refs from v1.0.0 to v1.0.1 and update copyright years (#2357)
2021-02-25 13:37:22 +00:00
# https://github.com/ethereum/eth2.0-specs/blob/v1.0.1/specs/phase0/beacon-chain.md#get_beacon_committee
misc memory and perf fixes (#1899) * misc memory and perf fixes * use EpochRef for attestation aggregation * compress effective balances in memory (medalla unfinalized: 4gb -> 1gb) * avoid hitting db when rewinding to head or clearance state * avoid hitting db when blocks can be applied to in-memory state - speeds up startup considerably * avoid storing epochref in fork choice * simplify and speed up beacon block creation flow - avoids state reload thanks to head rewind optimization * iterator-based committee and attestation participation help avoid lots of small memory allocations throughout epoch transition (40% speedup on epoch processing, for example during startup) * add constant for threshold
2020-10-22 10:53:33 +00:00
func get_beacon_committee_len*(
epochRef: EpochRef, slot: Slot, index: CommitteeIndex): uint64 =
# Return the number of members in the beacon committee at ``slot`` for ``index``.
let
committees_per_slot = get_committee_count_per_slot(epochRef)
compute_committee_len(
count_active_validators(epochRef),
(slot mod SLOTS_PER_EPOCH) * committees_per_slot +
index.uint64,
committees_per_slot * SLOTS_PER_EPOCH
)
bump spec refs from v1.0.0 to v1.0.1 and update copyright years (#2357)
2021-02-25 13:37:22 +00:00
# https://github.com/ethereum/eth2.0-specs/blob/v1.0.1/specs/phase0/beacon-chain.md#get_attesting_indices
misc memory and perf fixes (#1899) * misc memory and perf fixes * use EpochRef for attestation aggregation * compress effective balances in memory (medalla unfinalized: 4gb -> 1gb) * avoid hitting db when rewinding to head or clearance state * avoid hitting db when blocks can be applied to in-memory state - speeds up startup considerably * avoid storing epochref in fork choice * simplify and speed up beacon block creation flow - avoids state reload thanks to head rewind optimization * iterator-based committee and attestation participation help avoid lots of small memory allocations throughout epoch transition (40% speedup on epoch processing, for example during startup) * add constant for threshold
2020-10-22 10:53:33 +00:00
iterator get_attesting_indices*(epochRef: EpochRef,
data: AttestationData,
bits: CommitteeValidatorsBits):
ValidatorIndex =
if bits.lenu64 != get_beacon_committee_len(epochRef, data.slot, data.index.CommitteeIndex):
trace "get_attesting_indices: inconsistent aggregation and committee length"
else:
var i = 0
for index in get_beacon_committee(epochRef, data.slot, data.index.CommitteeIndex):
if bits[i]:
yield index
inc i
remove attestation/aggregate queue (#2519) With the introduction of batching and lazy attestation aggregation, it no longer makes sense to enqueue attestations between the signature check and adding them to the attestation pool - this only takes up valuable CPU without any real benefit. * add successfully validated attestations to attestion pool directly * avoid copying participant list around for single-vote attestations, pass single validator index instead * release decompressed gossip memory earlier, specially during async message validation * use cooked signatures in a few more places to avoid reloads and errors * remove some Defect-raising versions of signature-loading * release decompressed data memory before validating message
2021-04-26 20:39:44 +00:00
func get_attesting_indices_one*(epochRef: EpochRef,
data: AttestationData,
bits: CommitteeValidatorsBits):
Option[ValidatorIndex] =
# A variation on get_attesting_indices that returns the validator index only
# if only one validator index is set
if bits.lenu64 != get_beacon_committee_len(epochRef, data.slot, data.index.CommitteeIndex):
trace "get_attesting_indices: inconsistent aggregation and committee length"
none(ValidatorIndex)
else:
var res = none(ValidatorIndex)
var i = 0
for index in get_beacon_committee(epochRef, data.slot, data.index.CommitteeIndex):
if bits[i]:
if res.isNone():
res = some(index)
else:
return none(ValidatorIndex)
inc i
res
bump spec refs from v1.0.0 to v1.0.1 and update copyright years (#2357)
2021-02-25 13:37:22 +00:00
# https://github.com/ethereum/eth2.0-specs/blob/v1.0.1/specs/phase0/beacon-chain.md#get_attesting_indices
removed the BlockPool type and all of the proxy functions around it (#1401) * removed the BlockPool type and all of the proxy functions around it - passing the chain DAG and the quarantine explicitly where appropriately - they don't need to be bundled in a type * fixed the build after the rebase
2020-07-30 19:18:17 +00:00
func get_attesting_indices*(epochRef: EpochRef,
data: AttestationData,
bits: CommitteeValidatorsBits):
remove several IntSet usages in lieu of seq[ValidatorIndex] (#2288) * remove several IntSet usages in lieu of seq[ValidatorIndex] * convert smaller types to larger types * larger type, again
2021-02-08 07:27:30 +00:00
seq[ValidatorIndex] =
# TODO sequtils2 mapIt
misc memory and perf fixes (#1899) * misc memory and perf fixes * use EpochRef for attestation aggregation * compress effective balances in memory (medalla unfinalized: 4gb -> 1gb) * avoid hitting db when rewinding to head or clearance state * avoid hitting db when blocks can be applied to in-memory state - speeds up startup considerably * avoid storing epochref in fork choice * simplify and speed up beacon block creation flow - avoids state reload thanks to head rewind optimization * iterator-based committee and attestation participation help avoid lots of small memory allocations throughout epoch transition (40% speedup on epoch processing, for example during startup) * add constant for threshold
2020-10-22 10:53:33 +00:00
for idx in get_attesting_indices(epochRef, data, bits):
remove several IntSet usages in lieu of seq[ValidatorIndex] (#2288) * remove several IntSet usages in lieu of seq[ValidatorIndex] * convert smaller types to larger types * larger type, again
2021-02-08 07:27:30 +00:00
result.add(idx)
removed the BlockPool type and all of the proxy functions around it (#1401) * removed the BlockPool type and all of the proxy functions around it - passing the chain DAG and the quarantine explicitly where appropriately - they don't need to be bundled in a type * fixed the build after the rebase
2020-07-30 19:18:17 +00:00
bump spec refs from v1.0.0 to v1.0.1 and update copyright years (#2357)
2021-02-25 13:37:22 +00:00
# https://github.com/ethereum/eth2.0-specs/blob/v1.0.1/specs/phase0/beacon-chain.md#get_indexed_attestation
removed the BlockPool type and all of the proxy functions around it (#1401) * removed the BlockPool type and all of the proxy functions around it - passing the chain DAG and the quarantine explicitly where appropriately - they don't need to be bundled in a type * fixed the build after the rebase
2020-07-30 19:18:17 +00:00
func get_indexed_attestation*(epochRef: EpochRef, attestation: Attestation): IndexedAttestation =
# Return the indexed attestation corresponding to ``attestation``.
let
attesting_indices =
get_attesting_indices(
epochRef, attestation.data, attestation.aggregation_bits)
IndexedAttestation(
attesting_indices:
List[uint64, Limit MAX_VALIDATORS_PER_COMMITTEE].init(
sorted(mapIt(attesting_indices, it.uint64), system.cmp)),
data: attestation.data,
signature: attestation.signature
)
remove one cache, add another (#1449) * remove one cache, add another This cache removes the need for rewinding in most attestation validation flow since the attestations come from one of two epochs and must be targetting a viable block. Additionally, it also removes all state caches which are less likely to be used over-all - more metrics are needed to track the rewinding. On risk is that when chains don't finalize, we'll have lots of epochrefs in memory meaning lots of validator key databases, most being exactly the same. This can be addressed in any number of ways. Some of the memory usage is mitigated by the fact that we previously had lots of big state caches and now we're keeping only keys instead. * cleanups * doc
2020-08-06 19:48:47 +00:00
bump spec refs from v1.0.0 to v1.0.1 and update copyright years (#2357)
2021-02-25 13:37:22 +00:00
# https://github.com/ethereum/eth2.0-specs/blob/v1.0.1/specs/phase0/beacon-chain.md#is_valid_indexed_attestation
remove one cache, add another (#1449) * remove one cache, add another This cache removes the need for rewinding in most attestation validation flow since the attestations come from one of two epochs and must be targetting a viable block. Additionally, it also removes all state caches which are less likely to be used over-all - more metrics are needed to track the rewinding. On risk is that when chains don't finalize, we'll have lots of epochrefs in memory meaning lots of validator key databases, most being exactly the same. This can be addressed in any number of ways. Some of the memory usage is mitigated by the fact that we previously had lots of big state caches and now we're keeping only keys instead. * cleanups * doc
2020-08-06 19:48:47 +00:00
proc is_valid_indexed_attestation*(
fork: Fork, genesis_validators_root: Eth2Digest,
epochRef: EpochRef, indexed_attestation: SomeIndexedAttestation,
small spec cleanups (#1501) * clean up logging a bit * return error on indexed attestation check
2020-08-13 13:47:06 +00:00
flags: UpdateFlags): Result[void, cstring] =
remove one cache, add another (#1449) * remove one cache, add another This cache removes the need for rewinding in most attestation validation flow since the attestations come from one of two epochs and must be targetting a viable block. Additionally, it also removes all state caches which are less likely to be used over-all - more metrics are needed to track the rewinding. On risk is that when chains don't finalize, we'll have lots of epochrefs in memory meaning lots of validator key databases, most being exactly the same. This can be addressed in any number of ways. Some of the memory usage is mitigated by the fact that we previously had lots of big state caches and now we're keeping only keys instead. * cleanups * doc
2020-08-06 19:48:47 +00:00
# Check if ``indexed_attestation`` is not empty, has sorted and unique
# indices and has a valid aggregate signature.
template is_sorted_and_unique(s: untyped): bool =
small spec cleanups (#1501) * clean up logging a bit * return error on indexed attestation check
2020-08-13 13:47:06 +00:00
var res = true
remove one cache, add another (#1449) * remove one cache, add another This cache removes the need for rewinding in most attestation validation flow since the attestations come from one of two epochs and must be targetting a viable block. Additionally, it also removes all state caches which are less likely to be used over-all - more metrics are needed to track the rewinding. On risk is that when chains don't finalize, we'll have lots of epochrefs in memory meaning lots of validator key databases, most being exactly the same. This can be addressed in any number of ways. Some of the memory usage is mitigated by the fact that we previously had lots of big state caches and now we're keeping only keys instead. * cleanups * doc
2020-08-06 19:48:47 +00:00
for i in 1 ..< s.len:
if s[i - 1].uint64 >= s[i].uint64:
small spec cleanups (#1501) * clean up logging a bit * return error on indexed attestation check
2020-08-13 13:47:06 +00:00
res = false
break
res
remove one cache, add another (#1449) * remove one cache, add another This cache removes the need for rewinding in most attestation validation flow since the attestations come from one of two epochs and must be targetting a viable block. Additionally, it also removes all state caches which are less likely to be used over-all - more metrics are needed to track the rewinding. On risk is that when chains don't finalize, we'll have lots of epochrefs in memory meaning lots of validator key databases, most being exactly the same. This can be addressed in any number of ways. Some of the memory usage is mitigated by the fact that we previously had lots of big state caches and now we're keeping only keys instead. * cleanups * doc
2020-08-06 19:48:47 +00:00
small spec cleanups (#1501) * clean up logging a bit * return error on indexed attestation check
2020-08-13 13:47:06 +00:00
if len(indexed_attestation.attesting_indices) == 0:
return err("indexed_attestation: no attesting indices")
remove one cache, add another (#1449) * remove one cache, add another This cache removes the need for rewinding in most attestation validation flow since the attestations come from one of two epochs and must be targetting a viable block. Additionally, it also removes all state caches which are less likely to be used over-all - more metrics are needed to track the rewinding. On risk is that when chains don't finalize, we'll have lots of epochrefs in memory meaning lots of validator key databases, most being exactly the same. This can be addressed in any number of ways. Some of the memory usage is mitigated by the fact that we previously had lots of big state caches and now we're keeping only keys instead. * cleanups * doc
2020-08-06 19:48:47 +00:00
# Not from spec, but this function gets used in front-line roles, not just
# behind firewall.
let num_validators = epochRef.validator_keys.lenu64
if anyIt(indexed_attestation.attesting_indices, it >= num_validators):
small spec cleanups (#1501) * clean up logging a bit * return error on indexed attestation check
2020-08-13 13:47:06 +00:00
return err("indexed attestation: not all indices valid validators")
remove one cache, add another (#1449) * remove one cache, add another This cache removes the need for rewinding in most attestation validation flow since the attestations come from one of two epochs and must be targetting a viable block. Additionally, it also removes all state caches which are less likely to be used over-all - more metrics are needed to track the rewinding. On risk is that when chains don't finalize, we'll have lots of epochrefs in memory meaning lots of validator key databases, most being exactly the same. This can be addressed in any number of ways. Some of the memory usage is mitigated by the fact that we previously had lots of big state caches and now we're keeping only keys instead. * cleanups * doc
2020-08-06 19:48:47 +00:00
small spec cleanups (#1501) * clean up logging a bit * return error on indexed attestation check
2020-08-13 13:47:06 +00:00
if not is_sorted_and_unique(indexed_attestation.attesting_indices):
return err("indexed attestation: indices not sorted and unique")
remove one cache, add another (#1449) * remove one cache, add another This cache removes the need for rewinding in most attestation validation flow since the attestations come from one of two epochs and must be targetting a viable block. Additionally, it also removes all state caches which are less likely to be used over-all - more metrics are needed to track the rewinding. On risk is that when chains don't finalize, we'll have lots of epochrefs in memory meaning lots of validator key databases, most being exactly the same. This can be addressed in any number of ways. Some of the memory usage is mitigated by the fact that we previously had lots of big state caches and now we're keeping only keys instead. * cleanups * doc
2020-08-06 19:48:47 +00:00
# Verify aggregate signature
avoid copying keys for trusted signatures this removes `is_valid_indexed_attestion` from performance benchmarks when replaying blocks
2021-01-25 16:09:31 +00:00
if not (skipBLSValidation in flags or indexed_attestation.signature is TrustedSig):
small spec cleanups (#1501) * clean up logging a bit * return error on indexed attestation check
2020-08-13 13:47:06 +00:00
let pubkeys = mapIt(
indexed_attestation.attesting_indices, epochRef.validator_keys[it])
remove one cache, add another (#1449) * remove one cache, add another This cache removes the need for rewinding in most attestation validation flow since the attestations come from one of two epochs and must be targetting a viable block. Additionally, it also removes all state caches which are less likely to be used over-all - more metrics are needed to track the rewinding. On risk is that when chains don't finalize, we'll have lots of epochrefs in memory meaning lots of validator key databases, most being exactly the same. This can be addressed in any number of ways. Some of the memory usage is mitigated by the fact that we previously had lots of big state caches and now we're keeping only keys instead. * cleanups * doc
2020-08-06 19:48:47 +00:00
if not verify_attestation_signature(
fork, genesis_validators_root, indexed_attestation.data,
pubkeys, indexed_attestation.signature):
small spec cleanups (#1501) * clean up logging a bit * return error on indexed attestation check
2020-08-13 13:47:06 +00:00
return err("indexed attestation: signature verification failure")
implement clock disparity for attestation validation (#1568) This implements disparity, resolving a part of https://github.com/status-im/nim-beacon-chain/issues/1367 * make BeaconTime a duration for fractional seconds * factor out attestation/aggregate validation * simplify recording of queued attestations * simplify attestation signature check * fix blocks_received metric * add some trivial validation tests * remove unresolved attestation table - attestations for unknown blocks are dropped instead (cannot verify their signature)
2020-08-27 07:34:12 +00:00
ok()
bump spec refs from v1.0.0 to v1.0.1 and update copyright years (#2357)
2021-02-25 13:37:22 +00:00
# https://github.com/ethereum/eth2.0-specs/blob/v1.0.1/specs/phase0/beacon-chain.md#is_valid_indexed_attestation
implement clock disparity for attestation validation (#1568) This implements disparity, resolving a part of https://github.com/status-im/nim-beacon-chain/issues/1367 * make BeaconTime a duration for fractional seconds * factor out attestation/aggregate validation * simplify recording of queued attestations * simplify attestation signature check * fix blocks_received metric * add some trivial validation tests * remove unresolved attestation table - attestations for unknown blocks are dropped instead (cannot verify their signature)
2020-08-27 07:34:12 +00:00
proc is_valid_indexed_attestation*(
fork: Fork, genesis_validators_root: Eth2Digest,
attestation processing speedups * avoid creating indexed attestation just to check signatures - above all, don't create it when not checking signatures ;) * avoid pointer op when adding attestation to pool * better iterator for yielding attestations * add metric / log for attestation packing time
2021-04-14 14:43:29 +00:00
epochRef: EpochRef,
implement clock disparity for attestation validation (#1568) This implements disparity, resolving a part of https://github.com/status-im/nim-beacon-chain/issues/1367 * make BeaconTime a duration for fractional seconds * factor out attestation/aggregate validation * simplify recording of queued attestations * simplify attestation signature check * fix blocks_received metric * add some trivial validation tests * remove unresolved attestation table - attestations for unknown blocks are dropped instead (cannot verify their signature)
2020-08-27 07:34:12 +00:00
attestation: SomeAttestation, flags: UpdateFlags): Result[void, cstring] =
# This is a variation on `is_valid_indexed_attestation` that works directly
# with an attestation instead of first constructing an `IndexedAttestation`
# and then validating it - for the purpose of validating the signature, the
# order doesn't matter and we can proceed straight to validating the
# signature instead
attestation processing speedups * avoid creating indexed attestation just to check signatures - above all, don't create it when not checking signatures ;) * avoid pointer op when adding attestation to pool * better iterator for yielding attestations * add metric / log for attestation packing time
2021-04-14 14:43:29 +00:00
let sigs = attestation.aggregation_bits.countOnes()
if sigs == 0:
return err("is_valid_indexed_attestation: no attesting indices")
implement clock disparity for attestation validation (#1568) This implements disparity, resolving a part of https://github.com/status-im/nim-beacon-chain/issues/1367 * make BeaconTime a duration for fractional seconds * factor out attestation/aggregate validation * simplify recording of queued attestations * simplify attestation signature check * fix blocks_received metric * add some trivial validation tests * remove unresolved attestation table - attestations for unknown blocks are dropped instead (cannot verify their signature)
2020-08-27 07:34:12 +00:00
# Verify aggregate signature
avoid copying keys for trusted signatures this removes `is_valid_indexed_attestion` from performance benchmarks when replaying blocks
2021-01-25 16:09:31 +00:00
if not (skipBLSValidation in flags or attestation.signature is TrustedSig):
attestation processing speedups * avoid creating indexed attestation just to check signatures - above all, don't create it when not checking signatures ;) * avoid pointer op when adding attestation to pool * better iterator for yielding attestations * add metric / log for attestation packing time
2021-04-14 14:43:29 +00:00
var
pubkeys = newSeqOfCap[ValidatorPubKey](sigs)
for index in get_attesting_indices(
epochRef, attestation.data, attestation.aggregation_bits):
pubkeys.add(epochRef.validator_keys[index])
implement clock disparity for attestation validation (#1568) This implements disparity, resolving a part of https://github.com/status-im/nim-beacon-chain/issues/1367 * make BeaconTime a duration for fractional seconds * factor out attestation/aggregate validation * simplify recording of queued attestations * simplify attestation signature check * fix blocks_received metric * add some trivial validation tests * remove unresolved attestation table - attestations for unknown blocks are dropped instead (cannot verify their signature)
2020-08-27 07:34:12 +00:00
if not verify_attestation_signature(
fork, genesis_validators_root, attestation.data,
pubkeys, attestation.signature):
attestation processing speedups * avoid creating indexed attestation just to check signatures - above all, don't create it when not checking signatures ;) * avoid pointer op when adding attestation to pool * better iterator for yielding attestations * add metric / log for attestation packing time
2021-04-14 14:43:29 +00:00
return err("is_valid_indexed_attestation: signature verification failure")
remove one cache, add another (#1449) * remove one cache, add another This cache removes the need for rewinding in most attestation validation flow since the attestations come from one of two epochs and must be targetting a viable block. Additionally, it also removes all state caches which are less likely to be used over-all - more metrics are needed to track the rewinding. On risk is that when chains don't finalize, we'll have lots of epochrefs in memory meaning lots of validator key databases, most being exactly the same. This can be addressed in any number of ways. Some of the memory usage is mitigated by the fact that we previously had lots of big state caches and now we're keeping only keys instead. * cleanups * doc
2020-08-06 19:48:47 +00:00
small spec cleanups (#1501) * clean up logging a bit * return error on indexed attestation check
2020-08-13 13:47:06 +00:00
ok()
use epochcache for attesting (#1478)
2020-08-10 13:21:31 +00:00
func makeAttestationData*(
epochRef: EpochRef, bs: BlockSlot,
Datatype cleanup (#1953) * clear up spec todo * test fix * remove unnecessary toSszType * type * one more
2020-11-04 21:52:47 +00:00
committee_index: CommitteeIndex): AttestationData =
use epochcache for attesting (#1478)
2020-08-10 13:21:31 +00:00
## Create an attestation / vote for the block `bs` using the
## data in `epochRef` to fill in the rest of the fields.
## `epochRef` is the epoch information corresponding to the `bs` advanced to
## the slot we're attesting to.
let
slot = bs.slot
current_epoch = slot.compute_epoch_at_slot()
epoch_boundary_slot = compute_start_slot_at_epoch(current_epoch)
epoch_boundary_block = bs.blck.atSlot(epoch_boundary_slot)
doAssert current_epoch == epochRef.epoch
bump spec refs from v1.0.0 to v1.0.1 and update copyright years (#2357)
2021-02-25 13:37:22 +00:00
# https://github.com/ethereum/eth2.0-specs/blob/v1.0.1/specs/phase0/validator.md#attestation-data
use epochcache for attesting (#1478)
2020-08-10 13:21:31 +00:00
AttestationData(
slot: slot,
Datatype cleanup (#1953) * clear up spec todo * test fix * remove unnecessary toSszType * type * one more
2020-11-04 21:52:47 +00:00
index: committee_index.uint64,
use epochcache for attesting (#1478)
2020-08-10 13:21:31 +00:00
beacon_block_root: bs.blck.root,
source: epochRef.current_justified_checkpoint,
target: Checkpoint(
epoch: current_epoch,
root: epoch_boundary_block.blck.root
)
)
use epochref for get_committee_assignments() to avoid repeated shuffling (#2463) * use epochref for get_committee_assignments() to avoid repeated shuffling * remove unnecessary imports and StateCache() construction
2021-03-30 15:01:47 +00:00
# https://github.com/ethereum/eth2.0-specs/blob/v1.0.1/specs/phase0/validator.md#validator-assignments
iterator get_committee_assignments*(
epochRef: EpochRef, epoch: Epoch, validator_indices: IntSet):
tuple[validatorIndices: IntSet,
committeeIndex: CommitteeIndex,
Perform attestation check before broadcast (#2550) Currently, we have a bit of a convoluted flow where when sending attestations, we start broadcasting them over gossip then pass them to the attestation validation to include them in the local attestation pool - it should be the other way around: we should be checking attestations _before_ gossipping them - this serves as an additional safety net to ensure that we don't publish junk - this becomes more important when publishing attestations from the API. Also, the REST API was performing its own validation meaning attestations coming from REST would be validated twice - finally, the JSON RPC wasn't pre-validating and would happily broadcast invalid attestations. * Unified attestation production pipeline with the same flow for gossip, locally and API-produced attestations: all are now validated and entered into the pool, then broadcast/republished * Refactor subnet handling with specific SubnetId alias, streamlining where subnets are computed, avoiding the need to pass around the number of active validators * Move some of the subnet handling code to eth2_network * Use BitArray throughout for subnet handling
2021-05-10 07:13:36 +00:00
subnet_id: SubnetId, slot: Slot] =
use epochref for get_committee_assignments() to avoid repeated shuffling (#2463) * use epochref for get_committee_assignments() to avoid repeated shuffling * remove unnecessary imports and StateCache() construction
2021-03-30 15:01:47 +00:00
let
committees_per_slot = get_committee_count_per_slot(epochRef)
start_slot = compute_start_slot_at_epoch(epoch)
for slot in start_slot ..< start_slot + SLOTS_PER_EPOCH:
for index in 0'u64 ..< committees_per_slot:
let
idx = index.CommitteeIndex
includedIndices =
toIntSet(get_beacon_committee(epochRef, slot, idx)) *
validator_indices
if includedIndices.len > 0:
yield (
includedIndices, idx,
Perform attestation check before broadcast (#2550) Currently, we have a bit of a convoluted flow where when sending attestations, we start broadcasting them over gossip then pass them to the attestation validation to include them in the local attestation pool - it should be the other way around: we should be checking attestations _before_ gossipping them - this serves as an additional safety net to ensure that we don't publish junk - this becomes more important when publishing attestations from the API. Also, the REST API was performing its own validation meaning attestations coming from REST would be validated twice - finally, the JSON RPC wasn't pre-validating and would happily broadcast invalid attestations. * Unified attestation production pipeline with the same flow for gossip, locally and API-produced attestations: all are now validated and entered into the pool, then broadcast/republished * Refactor subnet handling with specific SubnetId alias, streamlining where subnets are computed, avoiding the need to pass around the number of active validators * Move some of the subnet handling code to eth2_network * Use BitArray throughout for subnet handling
2021-05-10 07:13:36 +00:00
compute_subnet_for_attestation(committees_per_slot, slot, idx),
use epochref for get_committee_assignments() to avoid repeated shuffling (#2463) * use epochref for get_committee_assignments() to avoid repeated shuffling * remove unnecessary imports and StateCache() construction
2021-03-30 15:01:47 +00:00
slot)