status-im
/
nimbus-eth2
mirror of https://github.com/status-im/nimbus-eth2.git
2
0
Fork 0
Code Issues Projects Releases Wiki Activity
nimbus-eth2/beacon_chain/beacon_node.nim

88 lines
2.8 KiB
Nim
Raw Normal View History

Separate validator duties from the Beacon Node (#976) * Separate validator duties from the Beacon Node * remove duplicate MaxEmptySlotCOunt * imports * have beacnde properly compile
2020-05-06 13:23:45 +00:00
# beacon_chain
partially integrate eth1 merge changes (#2548) * partially integrate eth1 merge changes * use hexToSeqByte() and validate execution engine opaque transaction length * remove incorrect REST serialization code
2021-05-20 10:44:13 +00:00
# Copyright (c) 2018-2021 Status Research & Development GmbH
Separate validator duties from the Beacon Node (#976) * Separate validator duties from the Beacon Node * remove duplicate MaxEmptySlotCOunt * imports * have beacnde properly compile
2020-05-06 13:23:45 +00:00
# Licensed and distributed under either of
# * MIT license (license terms in the root directory or at https://opensource.org/licenses/MIT).
# * Apache v2 license (license terms in the root directory or at https://www.apache.org/licenses/LICENSE-2.0).
# at your option. This file may not be copied, modified, or distributed except according to those terms.
import cleanup (#2997) * import cleanup ...and remove some unused types * add random imports * more imports
2021-10-19 14:09:26 +00:00
# Everything needed to run a full Beacon Node
Separate validator duties from the Beacon Node (#976) * Separate validator duties from the Beacon Node * remove duplicate MaxEmptySlotCOunt * imports * have beacnde properly compile
2020-05-06 13:23:45 +00:00
import
handle fork choice selecting a head unknown to chaindag (#1907) * move head slot metric to head selection * avoid redundant head selection in validator duties
2020-10-28 07:55:36 +00:00
std/osproc,
first batch of work towards the VC/BN split: - we have a new binary which connects via RPC to the respective BN and has an internal clock - waking it up on every slot - the BN has a new option called --external-validators and currently in order to have the VC binaries to run we need to pass EXTERNAL_VALIDATORS=yes to make - factored some code out of beacon_node.nim for easier reuse in validator_api.nim and validator_client.nim - the VC loads its associated private keys from the datadir for its BN - most of the validator API calls have been implemented as a stub. - the VC polls its BN at the start of each epoch - getting a list of all active validators for the current epoch - and then continues to request blocks and sign them with its appropriate validators when necessary
2020-05-22 17:04:52 +00:00
Separate validator duties from the Beacon Node (#976) * Separate validator duties from the Beacon Node * remove duplicate MaxEmptySlotCOunt * imports * have beacnde properly compile
2020-05-06 13:23:45 +00:00
# Nimble packages
Add a POST /api/nimbus/v1/graffiti API
2021-03-26 17:57:02 +00:00
chronos, json_rpc/servers/httpserver, presto,
Separate validator duties from the Beacon Node (#976) * Separate validator duties from the Beacon Node * remove duplicate MaxEmptySlotCOunt * imports * have beacnde properly compile
2020-05-06 13:23:45 +00:00
# Local modules
import cleanup (#2997) * import cleanup ...and remove some unused types * add random imports * more imports
2021-10-19 14:09:26 +00:00
"."/[beacon_clock, beacon_chain_db, conf],
clean up block processing (#2610) * gossip_to_consensus -> block_processor (it's processing only blocks, but not only from gossip) * measure queue and validation time for blocks * measure assignment and state loading times for updateStateData * avoid some unnecessary block copies in block sync * warn that database is corrupt if we hit tail without a state
2021-05-28 16:34:00 +00:00
./gossip_processing/[eth2_processor, block_processor, consensus_manager],
Reorg (5/5) (#2377) * Reorg things left into networking and gossip_processing * time -> beacon_clock * fix builds
2021-03-05 13:12:00 +00:00
./networking/eth2_network,
Move SSZ, deposit_contracts & eth1_monitor [reorg files 3/5] (#2371) * move deposit_contract * Move SSZ * fix ssz import in tests * move also eth1_monitor * forgot to delete the original * fix comma [skip ci] * Fix "make" & tools imports * Fix import * Fix import again * rename deposit_contract -> eth1 * Revert ssz move to subfolder * path fixes [skip ci]
2021-03-03 06:23:05 +00:00
./eth1/eth1_monitor,
clean up fork enum and field names * single naming strategy * simplify some fork code * simplify forked block production
2021-10-18 16:37:27 +00:00
./consensus_object_pools/[
import cleanup (#2997) * import cleanup ...and remove some unused types * add random imports * more imports
2021-10-19 14:09:26 +00:00
blockchain_dag, block_quarantine, exit_pool, attestation_pool,
sync_committee_msg_pool],
remove false OnBlockAdded dependency on phase0 HashedBeaconState (#2661) * remove false OnBlockAdded dependency on phase.HashedBeaconState * introduce altair data types into block_clearance; update some alpha.6 spec refs to alpha.7; add get_active_validator_indices_len ForkedHashedBeaconState wrapper * switch many modules from using datatypes (with phase0 states/blocks) to datatypes/base (fork-independent); update spec refs from alpha.6 to alpha.7 and remove rm'd G2_POINT_AT_INFINITY * switch more modules from using datatypes (with phase0 states/blocks) to datatypes/base (fork-independent); update spec refs from alpha.6 to alpha.7 * remove unnecessary phase0-only wrapper of get_attesting_indices(); allow signatures_batch to process either fork; remove O(n^2) nested loop in process_inactivity_updates(); add altair support to getAttestationsforTestBlock() * add Altair versions of asSigVerified(), asTrusted(), and makeBeaconBlock() * fix spec URL to be Altair for Altair makeBeaconBlock()
2021-06-21 08:35:24 +00:00
./spec/datatypes/base,
register vc duties with subnet tracker (#2949) * register vc duties with subnet tracker * fix activation logging during startup * cache slot signature to avoid duplicate signature work * schedule aggregation duties one slot at a time to avoid CPU spike at each epoch * lower aggregation subnet pre-subscription time to 4 slots (lowers bandwidth and CPU usage) * update stability subnets in ENR on startup * log gossip state * perform gossip subscriptions just before the next slot starts * document stuff * add random include * don't overwrite subscription state when not subscribed * log target gossip state * updating gossip status once is enough * add test * remove syncQueueLen - this one is not updated at the end of the sync and may cause gossip to disconnect itself completely - use a simple head distance instead * fix gossip disconnection - if in hysteresis, node.gossipState will be set to disabled even though we don't disable topic subscriptions * fix extra duty registration call
2021-10-18 09:11:44 +00:00
./sync/[sync_manager, request_manager],
Validator monitoring (#2925) Validator monitoring based on and mostly compatible with the implementation in Lighthouse - tracks additional logs and metrics for specified validators so as to stay on top on performance. The implementation works more or less the following way: * Validator pubkeys are singled out for monitoring - these can be running on the node or not * For every action that the validator takes, we record steps in the process such as messages being seen on the network or published in the API * When the dust settles at the end of an epoch, we report the information from one epoch before that, which coincides with the balances being updated - this is a tradeoff between being correct (waiting for finalization) and providing relevant information in a timely manner)
2021-12-20 19:20:31 +00:00
./validators/[action_tracker, validator_monitor, validator_pool]
Separate validator duties from the Beacon Node (#976) * Separate validator duties from the Beacon Node * remove duplicate MaxEmptySlotCOunt * imports * have beacnde properly compile
2020-05-06 13:23:45 +00:00
handle fork choice selecting a head unknown to chaindag (#1907) * move head slot metric to head selection * avoid redundant head selection in validator duties
2020-10-28 07:55:36 +00:00
export
register vc duties with subnet tracker (#2949) * register vc duties with subnet tracker * fix activation logging during startup * cache slot signature to avoid duplicate signature work * schedule aggregation duties one slot at a time to avoid CPU spike at each epoch * lower aggregation subnet pre-subscription time to 4 slots (lowers bandwidth and CPU usage) * update stability subnets in ENR on startup * log gossip state * perform gossip subscriptions just before the next slot starts * document stuff * add random include * don't overwrite subscription state when not subscribed * log target gossip state * updating gossip status once is enough * add test * remove syncQueueLen - this one is not updated at the end of the sync and may cause gossip to disconnect itself completely - use a simple head distance instead * fix gossip disconnection - if in hysteresis, node.gossipState will be set to disabled even though we don't disable topic subscriptions * fix extra duty registration call
2021-10-18 09:11:44 +00:00
osproc, chronos, httpserver, presto, action_tracker, beacon_clock,
import cleanup (#2997) * import cleanup ...and remove some unused types * add random imports * more imports
2021-10-19 14:09:26 +00:00
beacon_chain_db, conf, attestation_pool, sync_committee_msg_pool,
validator_pool, eth2_network, eth1_monitor, request_manager, sync_manager,
Validator monitoring (#2925) Validator monitoring based on and mostly compatible with the implementation in Lighthouse - tracks additional logs and metrics for specified validators so as to stay on top on performance. The implementation works more or less the following way: * Validator pubkeys are singled out for monitoring - these can be running on the node or not * For every action that the validator takes, we record steps in the process such as messages being seen on the network or published in the API * When the dust settles at the end of an epoch, we report the information from one epoch before that, which coincides with the balances being updated - this is a tradeoff between being correct (waiting for finalization) and providing relevant information in a timely manner)
2021-12-20 19:20:31 +00:00
eth2_processor, blockchain_dag, block_quarantine, base, exit_pool,
validator_monitor
first batch of work towards the VC/BN split: - we have a new binary which connects via RPC to the respective BN and has an internal clock - waking it up on every slot - the BN has a new option called --external-validators and currently in order to have the VC binaries to run we need to pass EXTERNAL_VALIDATORS=yes to make - factored some code out of beacon_node.nim for easier reuse in validator_api.nim and validator_client.nim - the VC loads its associated private keys from the datadir for its BN - most of the validator API calls have been implemented as a stub. - the VC polls its BN at the start of each epoch - getting a list of all active validators for the current epoch - and then continues to request blocks and sign them with its appropriate validators when necessary
2020-05-22 17:04:52 +00:00
Separate validator duties from the Beacon Node (#976) * Separate validator duties from the Beacon Node * remove duplicate MaxEmptySlotCOunt * imports * have beacnde properly compile
2020-05-06 13:23:45 +00:00
type
RpcServer* = RpcHttpServer
Merge fork gossip support (#3213) * Merge fork gossip support * index directly by BeaconStateFork and remove debugging log statement
2021-12-21 14:24:23 +00:00
GossipState* = set[BeaconStateFork]
altair topic switching (#2767) * altair topic switching * remove validate{Committee,Validator}IndexOr unused within branch
2021-08-09 12:54:45 +00:00
Separate validator duties from the Beacon Node (#976) * Separate validator duties from the Beacon Node * remove duplicate MaxEmptySlotCOunt * imports * have beacnde properly compile
2020-05-06 13:23:45 +00:00
BeaconNode* = ref object
nickname*: string
Add an option for graffiti customization
2020-06-29 17:30:19 +00:00
graffitiBytes*: GraffitiBytes
Separate validator duties from the Beacon Node (#976) * Separate validator duties from the Beacon Node * remove duplicate MaxEmptySlotCOunt * imports * have beacnde properly compile
2020-05-06 13:23:45 +00:00
network*: Eth2Node
Bump modules to take advantage of the new Json format flavors support Since quite a lot of additional procs were now compiled as generics, this lead to compiler bugs that had to be worked-around: * The `Domain` type was renamed to `Eth2Domain` to avoid compilation errors due to conflicts with `nativesockets.Domain`. Similarly, `eth2_network.KeyPair` was renamed to `NetKeyPair`. * A new more robust version of `hexToByteArray` was added to stew
2021-03-19 02:22:45 +00:00
netKeys*: NetKeyPair
Separate validator duties from the Beacon Node (#976) * Separate validator duties from the Beacon Node * remove duplicate MaxEmptySlotCOunt * imports * have beacnde properly compile
2020-05-06 13:23:45 +00:00
db*: BeaconChainDB
config*: BeaconNodeConf
startup cleanup * fix several memory leaks due to temporaries not being reset during init * avoid massive main() function with lots of stuff in it * disable nim-prompt (unused) * reuse validator pool instance in eth2_processor * style cleanup
2021-02-22 16:17:48 +00:00
attachedValidators*: ref ValidatorPool
singe validator key cache Instead of keeping a validator key list per EpochRef, this PR introduces a single shared validator key list in ChainDAG, and cleans up some other ChainDAG and key-related issues. The PR does not introduce the validator key list in the state transition - this is because we batch-check all signatures before entering the spec code, thus the spec code never hits the cache. A future refactor should _probably_ remove the threadvar altogether. There's a few other small fixes in here that make the flow easier to read: * fix `var ChainDAGRef` -> `ChainDAGRef` * fix `var QuarantineRef` -> `QuarantineRef` * consistent `dag` variable name * avoid using threadvar pubkey cache in most cases * better error messages in batch signature checking
2021-06-01 11:13:40 +00:00
dag*: ChainDAGRef
move quarantine outside of chaindag (#3124) * move quarantine outside of chaindag The quarantine has been part of the ChainDAG for the longest time, but this design has a few issues: * the function in which blocks are verified and added to the dag becomes reentrant and therefore difficult to reason about - we're currently using a stateful flag to work around it * quarantined blocks bypass the processing queue leading to a processing stampede * the quarantine flow is unsuitable for orphaned attestations - these should also should be quarantined eventually Instead of processing the quarantine inside ChainDAG, this PR moves re-queueing to `block_processor` which already is responsible for dealing with follow-up work when a block is added to the dag This sets the stage for keeping attestations in the quarantine as well. Also: * make `BlockError` `{.pure.}` * avoid use of `ValidationResult` in block clearance (that's for gossip)
2021-12-06 09:49:01 +00:00
quarantine*: ref Quarantine
avoid double deserialization When blocks and attestations arrive, they are SSZ-decoded twice: once for validation and once for processing. This branch enqueues the decoded block directly for processing, avoiding the second, slow deserialization. * move processing of blocks and attestations to queue * ...and out from beacon_node * split attestation processing into attestations and aggregates * also updates metrics * clean up logging to better follow the lifetime of gossip: arrival, validation and processing * drop attestations and aggregates if there are too many * try to prioritise blocks and aggregates before single-validator attestations
2020-08-20 16:30:47 +00:00
attestationPool*: ref AttestationPool
introduce SyncCommitteeMsgPool to eth2_processor and nimbus_beacon_node (#2831)
2021-08-28 22:27:51 +00:00
syncCommitteeMsgPool*: ref SyncCommitteeMsgPool
[WIP] skeleton of attester slashing pool & validators (#1639) * skeleton of attester slashing pool & validators * add skeleton for proposer slashings and voluntary exits; rename pool to more inclusive exit pool to stay consistent with all three; ensure is initialized by beacon_node so is safe to merge, even if it doesn't do much yet
2020-09-14 14:26:31 +00:00
exitPool*: ref ExitPool
Further reduce the number of RPC requests; Significantly faster ETH1 syncing
2020-11-03 01:21:07 +00:00
eth1Monitor*: Eth1Monitor
Separate validator duties from the Beacon Node (#976) * Separate validator duties from the Beacon Node * remove duplicate MaxEmptySlotCOunt * imports * have beacnde properly compile
2020-05-06 13:23:45 +00:00
rpcServer*: RpcServer
Rest API initial implementation.
2021-03-17 18:46:45 +00:00
restServer*: RestServerRef
Implementation of the Keymanager API (BETA) https://github.com/ethereum/keymanager-APIs
2021-12-22 12:37:31 +00:00
keymanagerServer*: RestServerRef
keymanagerToken*: Option[string]
REST /eth/v1/events API call implementation. (#2878) * Placing callbacks into strategic places. * Initial events call implementation. * Post rebase fixes. * Change addSyncContribution() implementation. * Add `attestation-sent` event. Remove gcsafe, raises from callbacks implementations. Move `attestation-received` fire at the end of attestation processing. * Address review comments.
2021-09-22 12:17:15 +00:00
eventBus*: AsyncEventBus
use a separate process for the private keys (Off by default) - there is a new signing_process binary which loads all validators of the beacon node and the BN dictates through stdin of the signing process what to be signed and when and reads from stdout of the process
2020-09-01 13:44:40 +00:00
vcProcess*: Process
Use one single async queue and loop for processing blocks. (#1487) * Initial commit * Fix compilation problem. * Address review comments.
2020-08-12 09:29:11 +00:00
requestManager*: RequestManager
Add proper concurrent connections. Add SeenTable to avoid continuous attempts to dead peers. Refactor onSecond. Block backward sync while forward sync is working. SyncManager now checks responses according corresponding requests + tests. SyncManager now watching for not progressing local_head_slot and resets SyncQueue.
2020-06-03 08:46:29 +00:00
syncManager*: SyncManager[Peer, PeerID]
Embed the Altona metadata in the NBC executable
2020-07-02 15:14:11 +00:00
genesisSnapshotContent*: string
register vc duties with subnet tracker (#2949) * register vc duties with subnet tracker * fix activation logging during startup * cache slot signature to avoid duplicate signature work * schedule aggregation duties one slot at a time to avoid CPU spike at each epoch * lower aggregation subnet pre-subscription time to 4 slots (lowers bandwidth and CPU usage) * update stability subnets in ENR on startup * log gossip state * perform gossip subscriptions just before the next slot starts * document stuff * add random include * don't overwrite subscription state when not subscribed * log target gossip state * updating gossip status once is enough * add test * remove syncQueueLen - this one is not updated at the end of the sync and may cause gossip to disconnect itself completely - use a simple head distance instead * fix gossip disconnection - if in hysteresis, node.gossipState will be set to disabled even though we don't disable topic subscriptions * fix extra duty registration call
2021-10-18 09:11:44 +00:00
actionTracker*: ActionTracker
avoid double deserialization When blocks and attestations arrive, they are SSZ-decoded twice: once for validation and once for processing. This branch enqueues the decoded block directly for processing, avoiding the second, slow deserialization. * move processing of blocks and attestations to queue * ...and out from beacon_node * split attestation processing into attestations and aggregates * also updates metrics * clean up logging to better follow the lifetime of gossip: arrival, validation and processing * drop attestations and aggregates if there are too many * try to prioritise blocks and aggregates before single-validator attestations
2020-08-20 16:30:47 +00:00
processor*: ref Eth2Processor
clean up block processing (#2610) * gossip_to_consensus -> block_processor (it's processing only blocks, but not only from gossip) * measure queue and validation time for blocks * measure assignment and state loading times for updateStateData * avoid some unnecessary block copies in block sync * warn that database is corrupt if we hit tail without a state
2021-05-28 16:34:00 +00:00
blockProcessor*: ref BlockProcessor
Split Eth2Processor in prep for batching (#2396) * Split Eth2Processor in gossip and consensus part and materialize the shared block queue * Update initialization in test_sync_manager
2021-03-11 10:10:57 +00:00
consensusManager*: ref ConsensusManager
Don't rely on a metric value for the ETH display in the status bar
2020-11-29 21:35:39 +00:00
attachedValidatorBalanceTotal*: uint64
altair topic switching (#2767) * altair topic switching * remove validate{Committee,Validator}IndexOr unused within branch
2021-08-09 12:54:45 +00:00
gossipState*: GossipState
move BeaconClock from ChainDAG to BeaconNode (#2796)
2021-08-20 08:58:15 +00:00
beaconClock*: BeaconClock
REST /eth/v1/events API call implementation. (#2878) * Placing callbacks into strategic places. * Initial events call implementation. * Post rebase fixes. * Change addSyncContribution() implementation. * Add `attestation-sent` event. Remove gcsafe, raises from callbacks implementations. Move `attestation-received` fire at the end of attestation processing. * Address review comments.
2021-09-22 12:17:15 +00:00
onAttestationSent*: OnAttestationCallback
Number of REST fixes. (#2987) * Initial commit. * Fix path. * Add validator keys to indices cache mechanism. Move syncComitteeParticipants to common place. * Fix sync participants order issue. * Fix error code when state could not be found. Refactor `state/validators` to use keysToIndices mechanism. * Fix RestValidatorIndex to ValidatorIndex conversion TODOs. * Address review comments. * Fix REST test rules.
2021-10-14 10:38:38 +00:00
restKeysCache*: Table[ValidatorPubKey, ValidatorIndex]
Validator monitoring (#2925) Validator monitoring based on and mostly compatible with the implementation in Lighthouse - tracks additional logs and metrics for specified validators so as to stay on top on performance. The implementation works more or less the following way: * Validator pubkeys are singled out for monitoring - these can be running on the node or not * For every action that the validator takes, we record steps in the process such as messages being seen on the network or published in the API * When the dust settles at the end of an epoch, we report the information from one epoch before that, which coincides with the balances being updated - this is a tradeoff between being correct (waiting for finalization) and providing relevant information in a timely manner)
2021-12-20 19:20:31 +00:00
validatorMonitor*: ref ValidatorMonitor
Separate validator duties from the Beacon Node (#976) * Separate validator duties from the Beacon Node * remove duplicate MaxEmptySlotCOunt * imports * have beacnde properly compile
2020-05-06 13:23:45 +00:00
first batch of work towards the VC/BN split: - we have a new binary which connects via RPC to the respective BN and has an internal clock - waking it up on every slot - the BN has a new option called --external-validators and currently in order to have the VC binaries to run we need to pass EXTERNAL_VALIDATORS=yes to make - factored some code out of beacon_node.nim for easier reuse in validator_api.nim and validator_client.nim - the VC loads its associated private keys from the datadir for its BN - most of the validator API calls have been implemented as a stub. - the VC polls its BN at the start of each epoch - getting a list of all active validators for the current epoch - and then continues to request blocks and sign them with its appropriate validators when necessary
2020-05-22 17:04:52 +00:00
const
MaxEmptySlotCount* = uint64(10*60) div SECONDS_PER_SLOT
Separate validator duties from the Beacon Node (#976) * Separate validator duties from the Beacon Node * remove duplicate MaxEmptySlotCOunt * imports * have beacnde properly compile
2020-05-06 13:23:45 +00:00
stop discarding future epochs; remove a StateCache() construction (#1610) * stop discarding non-existent future epochs during epoch state transitions; remove a pointless StateCache() construction in advance_slots() * update nbench to pass StateCache to process_slots()
2020-09-07 15:04:33 +00:00
# TODO stew/sequtils2
openarray -> openArray
2020-10-28 18:35:31 +00:00
template findIt*(s: openArray, predicate: untyped): int =
more work on the BN/VC split - fixed comments from the last review - getting more data VIA RPC, moved some code back into the BN only - attestation duties being requested as well
2020-05-27 17:06:28 +00:00
var res = -1
for i, it {.inject.} in s:
if predicate:
res = i
break
res
move rpc api to own folder, mimic upstream structure (#1905) also implements a few more endpoints
2020-10-27 09:00:57 +00:00
proc currentSlot*(node: BeaconNode): Slot =
node.beaconClock.now.slotOrZero