status-im
/
nimbus-eth2
mirror of https://github.com/status-im/nimbus-eth2.git
2
0
Fork 0
Code Issues Projects Releases Wiki Activity
nimbus-eth2/beacon_chain/beacon_node_common.nim

93 lines
2.8 KiB
Nim
Raw Normal View History

Separate validator duties from the Beacon Node (#976) * Separate validator duties from the Beacon Node * remove duplicate MaxEmptySlotCOunt * imports * have beacnde properly compile
2020-05-06 13:23:45 +00:00
# beacon_chain
# Copyright (c) 2018-2020 Status Research & Development GmbH
# Licensed and distributed under either of
# * MIT license (license terms in the root directory or at https://opensource.org/licenses/MIT).
# * Apache v2 license (license terms in the root directory or at https://www.apache.org/licenses/LICENSE-2.0).
# at your option. This file may not be copied, modified, or distributed except according to those terms.
# Common routines for a BeaconNode and a BeaconValidator node
import
first batch of work towards the VC/BN split: - we have a new binary which connects via RPC to the respective BN and has an internal clock - waking it up on every slot - the BN has a new option called --external-validators and currently in order to have the VC binaries to run we need to pass EXTERNAL_VALIDATORS=yes to make - factored some code out of beacon_node.nim for easier reuse in validator_api.nim and validator_client.nim - the VC loads its associated private keys from the datadir for its BN - most of the validator API calls have been implemented as a stub. - the VC polls its BN at the start of each epoch - getting a list of all active validators for the current epoch - and then continues to request blocks and sign them with its appropriate validators when necessary
2020-05-22 17:04:52 +00:00
# Standard library
- extracted the commands to run a VC into a separate run_validator.sh script - each BN gets half of its previous validators as inProcess and the other half goes to the respective VC for that BN - using separate data dirs where the keys are copied - also removed a few command line options which are no longer necessary - block proposals originating from a VC are propagated from one BN to the rest properly - other cleanup & moving code back to since it is no longer used elsewhere
2020-06-05 09:57:40 +00:00
tables,
first batch of work towards the VC/BN split: - we have a new binary which connects via RPC to the respective BN and has an internal clock - waking it up on every slot - the BN has a new option called --external-validators and currently in order to have the VC binaries to run we need to pass EXTERNAL_VALIDATORS=yes to make - factored some code out of beacon_node.nim for easier reuse in validator_api.nim and validator_client.nim - the VC loads its associated private keys from the datadir for its BN - most of the validator API calls have been implemented as a stub. - the VC polls its BN at the start of each epoch - getting a list of all active validators for the current epoch - and then continues to request blocks and sign them with its appropriate validators when necessary
2020-05-22 17:04:52 +00:00
Separate validator duties from the Beacon Node (#976) * Separate validator duties from the Beacon Node * remove duplicate MaxEmptySlotCOunt * imports * have beacnde properly compile
2020-05-06 13:23:45 +00:00
# Nimble packages
chronos, json_rpc/rpcserver, metrics,
first batch of work towards the VC/BN split: - we have a new binary which connects via RPC to the respective BN and has an internal clock - waking it up on every slot - the BN has a new option called --external-validators and currently in order to have the VC binaries to run we need to pass EXTERNAL_VALIDATORS=yes to make - factored some code out of beacon_node.nim for easier reuse in validator_api.nim and validator_client.nim - the VC loads its associated private keys from the datadir for its BN - most of the validator API calls have been implemented as a stub. - the VC polls its BN at the start of each epoch - getting a list of all active validators for the current epoch - and then continues to request blocks and sign them with its appropriate validators when necessary
2020-05-22 17:04:52 +00:00
chronicles,
Separate validator duties from the Beacon Node (#976) * Separate validator duties from the Beacon Node * remove duplicate MaxEmptySlotCOunt * imports * have beacnde properly compile
2020-05-06 13:23:45 +00:00
# Local modules
- extracted the commands to run a VC into a separate run_validator.sh script - each BN gets half of its previous validators as inProcess and the other half goes to the respective VC for that BN - using separate data dirs where the keys are copied - also removed a few command line options which are no longer necessary - block proposals originating from a VC are propagated from one BN to the rest properly - other cleanup & moving code back to since it is no longer used elsewhere
2020-06-05 09:57:40 +00:00
spec/[datatypes, crypto, digest],
conf, time, beacon_chain_db,
removed the BlockPool type and all of the proxy functions around it (#1401) * removed the BlockPool type and all of the proxy functions around it - passing the chain DAG and the quarantine explicitly where appropriately - they don't need to be bundled in a type * fixed the build after the rebase
2020-07-30 19:18:17 +00:00
attestation_pool, eth2_network,
renamed CandidateChains to ChainDagRef and made the Quarantine type a ref type so there is a single instance in the beacon node (#1407)
2020-07-31 14:49:06 +00:00
block_pools/[chain_dag, quarantine],
Add proper concurrent connections. Add SeenTable to avoid continuous attempts to dead peers. Refactor onSecond. Block backward sync while forward sync is working. SyncManager now checks responses according corresponding requests + tests. SyncManager now watching for not progressing local_head_slot and resets SyncQueue.
2020-06-03 08:46:29 +00:00
beacon_node_types, mainchain_monitor, request_manager,
Revert "Dual headed fork choice [Reloaded] (#1223)" (#1234) This reverts commit 6836d41ebda0ac2181adfd82489f67f968a44aa3.
2020-06-25 09:36:03 +00:00
sync_manager
Separate validator duties from the Beacon Node (#976) * Separate validator duties from the Beacon Node * remove duplicate MaxEmptySlotCOunt * imports * have beacnde properly compile
2020-05-06 13:23:45 +00:00
Fix some build problems and warnings
2020-05-25 17:33:03 +00:00
# This removes an invalid Nim warning that the digest module is unused here
# It's currently used for `shortLog(head.blck.root)`
type Eth2Digest = digest.Eth2Digest
first batch of work towards the VC/BN split: - we have a new binary which connects via RPC to the respective BN and has an internal clock - waking it up on every slot - the BN has a new option called --external-validators and currently in order to have the VC binaries to run we need to pass EXTERNAL_VALIDATORS=yes to make - factored some code out of beacon_node.nim for easier reuse in validator_api.nim and validator_client.nim - the VC loads its associated private keys from the datadir for its BN - most of the validator API calls have been implemented as a stub. - the VC polls its BN at the start of each epoch - getting a list of all active validators for the current epoch - and then continues to request blocks and sign them with its appropriate validators when necessary
2020-05-22 17:04:52 +00:00
Separate validator duties from the Beacon Node (#976) * Separate validator duties from the Beacon Node * remove duplicate MaxEmptySlotCOunt * imports * have beacnde properly compile
2020-05-06 13:23:45 +00:00
type
RpcServer* = RpcHttpServer
bump libp2p (#1267)
2020-07-01 11:41:40 +00:00
KeyPair* = eth2_network.KeyPair
Separate validator duties from the Beacon Node (#976) * Separate validator duties from the Beacon Node * remove duplicate MaxEmptySlotCOunt * imports * have beacnde properly compile
2020-05-06 13:23:45 +00:00
BeaconNode* = ref object
nickname*: string
Add an option for graffiti customization
2020-06-29 17:30:19 +00:00
graffitiBytes*: GraffitiBytes
Separate validator duties from the Beacon Node (#976) * Separate validator duties from the Beacon Node * remove duplicate MaxEmptySlotCOunt * imports * have beacnde properly compile
2020-05-06 13:23:45 +00:00
network*: Eth2Node
netKeys*: KeyPair
db*: BeaconChainDB
config*: BeaconNodeConf
attachedValidators*: ValidatorPool
renamed CandidateChains to ChainDagRef and made the Quarantine type a ref type so there is a single instance in the beacon node (#1407)
2020-07-31 14:49:06 +00:00
chainDag*: ChainDAGRef
quarantine*: QuarantineRef
Separate validator duties from the Beacon Node (#976) * Separate validator duties from the Beacon Node * remove duplicate MaxEmptySlotCOunt * imports * have beacnde properly compile
2020-05-06 13:23:45 +00:00
attestationPool*: AttestationPool
mainchainMonitor*: MainchainMonitor
beaconClock*: BeaconClock
rpcServer*: RpcServer
forkDigest*: ForkDigest
Use one single async queue and loop for processing blocks. (#1487) * Initial commit * Fix compilation problem. * Address review comments.
2020-08-12 09:29:11 +00:00
blocksQueue*: AsyncQueue[SyncBlock]
requestManager*: RequestManager
Add proper concurrent connections. Add SeenTable to avoid continuous attempts to dead peers. Refactor onSecond. Block backward sync while forward sync is working. SyncManager now checks responses according corresponding requests + tests. SyncManager now watching for not progressing local_head_slot and resets SyncQueue.
2020-06-03 08:46:29 +00:00
syncManager*: SyncManager[Peer, PeerID]
Separate validator duties from the Beacon Node (#976) * Separate validator duties from the Beacon Node * remove duplicate MaxEmptySlotCOunt * imports * have beacnde properly compile
2020-05-06 13:23:45 +00:00
topicBeaconBlocks*: string
topicAggregateAndProofs*: string
Use one single async queue and loop for processing blocks. (#1487) * Initial commit * Fix compilation problem. * Address review comments.
2020-08-12 09:29:11 +00:00
blockProcessingLoop*: Future[void]
Add proper concurrent connections. Add SeenTable to avoid continuous attempts to dead peers. Refactor onSecond. Block backward sync while forward sync is working. SyncManager now checks responses according corresponding requests + tests. SyncManager now watching for not progressing local_head_slot and resets SyncQueue.
2020-06-03 08:46:29 +00:00
onSecondLoop*: Future[void]
Embed the Altona metadata in the NBC executable
2020-07-02 15:14:11 +00:00
genesisSnapshotContent*: string
initial dynamic subscribe/unsubscribe for attestations to/from subnets (#1462) * initial dynamic subscribe/unsubscribe for attestations to/from subnets * implement random stability subnet and clean up * switch from HashSet[uint64] to set[uint8] * refactor subnet logic out from beacon_node and actual (un)subscribing * only try to subscribe to marginally different subnets * add assertions * maintain ENR subnets * assert that beacon_node and eth2_network have consistent view of subscribed subnets * disable actual cycling
2020-08-12 17:48:31 +00:00
attestationSubnets*: AttestationSubnets
Separate validator duties from the Beacon Node (#976) * Separate validator duties from the Beacon Node * remove duplicate MaxEmptySlotCOunt * imports * have beacnde properly compile
2020-05-06 13:23:45 +00:00
first batch of work towards the VC/BN split: - we have a new binary which connects via RPC to the respective BN and has an internal clock - waking it up on every slot - the BN has a new option called --external-validators and currently in order to have the VC binaries to run we need to pass EXTERNAL_VALIDATORS=yes to make - factored some code out of beacon_node.nim for easier reuse in validator_api.nim and validator_client.nim - the VC loads its associated private keys from the datadir for its BN - most of the validator API calls have been implemented as a stub. - the VC polls its BN at the start of each epoch - getting a list of all active validators for the current epoch - and then continues to request blocks and sign them with its appropriate validators when necessary
2020-05-22 17:04:52 +00:00
const
MaxEmptySlotCount* = uint64(10*60) div SECONDS_PER_SLOT
Separate validator duties from the Beacon Node (#976) * Separate validator duties from the Beacon Node * remove duplicate MaxEmptySlotCOunt * imports * have beacnde properly compile
2020-05-06 13:23:45 +00:00
# Metrics
declareGauge beacon_head_root,
"Root of the head block of the beacon chain"
Fork choice fixes 5 (#1381) * limit attestations kept in attestation pool With fork choice updated, the attestation pool only needs to keep track of attestations that will eventually end up in blocks - we can thus limit the horizon of attestations that we keep more aggressively. To get here, we expose getEpochRef which gets metadata about a particular epochref, and make sure to populate it when a block is added - this ensures that state rewinds during block addition are minimized. In addition, we'll use the target root/epoch when validating attestations - this helps minimize the number of different states that we need to rewind to, in general. * remove CandidateChains.justifiedState unused * remove BlockPools.Head object * avoid quadratic quarantine loop * fix
2020-07-28 13:54:32 +00:00
proc updateHead*(node: BeaconNode, wallSlot: Slot): BlockRef =
Separate validator duties from the Beacon Node (#976) * Separate validator duties from the Beacon Node * remove duplicate MaxEmptySlotCOunt * imports * have beacnde properly compile
2020-05-06 13:23:45 +00:00
# Check pending attestations - maybe we found some blocks for them
Fork choice fixes 5 (#1381) * limit attestations kept in attestation pool With fork choice updated, the attestation pool only needs to keep track of attestations that will eventually end up in blocks - we can thus limit the horizon of attestations that we keep more aggressively. To get here, we expose getEpochRef which gets metadata about a particular epochref, and make sure to populate it when a block is added - this ensures that state rewinds during block addition are minimized. In addition, we'll use the target root/epoch when validating attestations - this helps minimize the number of different states that we need to rewind to, in general. * remove CandidateChains.justifiedState unused * remove BlockPools.Head object * avoid quadratic quarantine loop * fix
2020-07-28 13:54:32 +00:00
node.attestationPool.resolve(wallSlot)
Separate validator duties from the Beacon Node (#976) * Separate validator duties from the Beacon Node * remove duplicate MaxEmptySlotCOunt * imports * have beacnde properly compile
2020-05-06 13:23:45 +00:00
# Grab the new head according to our latest attestation data
Fork choice fixes 5 (#1381) * limit attestations kept in attestation pool With fork choice updated, the attestation pool only needs to keep track of attestations that will eventually end up in blocks - we can thus limit the horizon of attestations that we keep more aggressively. To get here, we expose getEpochRef which gets metadata about a particular epochref, and make sure to populate it when a block is added - this ensures that state rewinds during block addition are minimized. In addition, we'll use the target root/epoch when validating attestations - this helps minimize the number of different states that we need to rewind to, in general. * remove CandidateChains.justifiedState unused * remove BlockPools.Head object * avoid quadratic quarantine loop * fix
2020-07-28 13:54:32 +00:00
let newHead = node.attestationPool.selectHead(wallSlot)
Separate validator duties from the Beacon Node (#976) * Separate validator duties from the Beacon Node * remove duplicate MaxEmptySlotCOunt * imports * have beacnde properly compile
2020-05-06 13:23:45 +00:00
removed the BlockPool type and all of the proxy functions around it (#1401) * removed the BlockPool type and all of the proxy functions around it - passing the chain DAG and the quarantine explicitly where appropriately - they don't need to be bundled in a type * fixed the build after the rebase
2020-07-30 19:18:17 +00:00
# Store the new head in the chain DAG - this may cause epochs to be
Separate validator duties from the Beacon Node (#976) * Separate validator duties from the Beacon Node * remove duplicate MaxEmptySlotCOunt * imports * have beacnde properly compile
2020-05-06 13:23:45 +00:00
# justified and finalized
removed the BlockPool type and all of the proxy functions around it (#1401) * removed the BlockPool type and all of the proxy functions around it - passing the chain DAG and the quarantine explicitly where appropriately - they don't need to be bundled in a type * fixed the build after the rebase
2020-07-30 19:18:17 +00:00
let oldFinalized = node.chainDag.finalizedHead.blck
Fork choice fixes 2 (#1356) * fork choice cleanup * enable v2 pruning * prefer `get_current_epoch` * fix finalization check to use correct epoch * small cleanups * add `count_active_validators` * remove misleading logs * fix justified checkpoint slot calculation in rpc
2020-07-22 21:01:44 +00:00
removed the BlockPool type and all of the proxy functions around it (#1401) * removed the BlockPool type and all of the proxy functions around it - passing the chain DAG and the quarantine explicitly where appropriately - they don't need to be bundled in a type * fixed the build after the rebase
2020-07-30 19:18:17 +00:00
node.chainDag.updateHead(newHead)
Separate validator duties from the Beacon Node (#976) * Separate validator duties from the Beacon Node * remove duplicate MaxEmptySlotCOunt * imports * have beacnde properly compile
2020-05-06 13:23:45 +00:00
beacon_head_root.set newHead.root.toGaugeValue
Dual headed fork choice [Revolution] (#1238) * Dual headed fork choice * fix finalizedEpoch not moving * reduce fork choice verbosity * Add failing tests due to pruning * Properly handle duplicate blocks in sync * test_block_pool also add a test for duplicate blocks * comments addressing review * Fix fork choice v2, was missing integrating block proposed * remove a spurious debug writeStackTrace * update block_sim * Use OrderedTable to ensure that we always load parents before children in fork choice * Load the DAG data in fork choice at init if there is some (can sync witti) * Cluster of quarantined blocks were not properly added to the fork choice * Workaround async gcsafe warnings * Update blockpoool tests * Do the callback before clearing the quarantine * Revert OrderedTable, implement topological sort of DAG, allow forkChoice to be initialized from arbitrary finalized heads * Make it work with latest devel - Altona readyness * Add a recovery mechanism when forkchoice desyncs with blockpool * add the current problematic node to the stack * Fix rebase indentation bug (but still producing invalid block) * Fix cache at epoch boundaries and lateBlock addition
2020-07-09 09:29:32 +00:00
# Cleanup the fork choice v2 if we have a finalized head
removed the BlockPool type and all of the proxy functions around it (#1401) * removed the BlockPool type and all of the proxy functions around it - passing the chain DAG and the quarantine explicitly where appropriately - they don't need to be bundled in a type * fixed the build after the rebase
2020-07-30 19:18:17 +00:00
if oldFinalized != node.chainDag.finalizedHead.blck:
Use fork v2 (#1358) * fork choice fixes, round 3 * introduce checkpoint tracker * split out fork choice backend that is independent of dag * correctly update best checkpoint to use for head selection * correctly consider wall clock when processing attestations * preload head history only (only one history is loaded from database anyway) * love the DAG * switch to fork choice v2 also remove BlockRef.children * fix
2020-07-25 19:41:12 +00:00
node.attestationPool.prune()
Dual headed fork choice [Revolution] (#1238) * Dual headed fork choice * fix finalizedEpoch not moving * reduce fork choice verbosity * Add failing tests due to pruning * Properly handle duplicate blocks in sync * test_block_pool also add a test for duplicate blocks * comments addressing review * Fix fork choice v2, was missing integrating block proposed * remove a spurious debug writeStackTrace * update block_sim * Use OrderedTable to ensure that we always load parents before children in fork choice * Load the DAG data in fork choice at init if there is some (can sync witti) * Cluster of quarantined blocks were not properly added to the fork choice * Workaround async gcsafe warnings * Update blockpoool tests * Do the callback before clearing the quarantine * Revert OrderedTable, implement topological sort of DAG, allow forkChoice to be initialized from arbitrary finalized heads * Make it work with latest devel - Altona readyness * Add a recovery mechanism when forkchoice desyncs with blockpool * add the current problematic node to the stack * Fix rebase indentation bug (but still producing invalid block) * Fix cache at epoch boundaries and lateBlock addition
2020-07-09 09:29:32 +00:00
Separate validator duties from the Beacon Node (#976) * Separate validator duties from the Beacon Node * remove duplicate MaxEmptySlotCOunt * imports * have beacnde properly compile
2020-05-06 13:23:45 +00:00
newHead
more work on the BN/VC split - fixed comments from the last review - getting more data VIA RPC, moved some code back into the BN only - attestation duties being requested as well
2020-05-27 17:06:28 +00:00
Fork choice fixes 2 (#1356) * fork choice cleanup * enable v2 pruning * prefer `get_current_epoch` * fix finalization check to use correct epoch * small cleanups * add `count_active_validators` * remove misleading logs * fix justified checkpoint slot calculation in rpc
2020-07-22 21:01:44 +00:00
template findIt*(s: openarray, predicate: untyped): int =
more work on the BN/VC split - fixed comments from the last review - getting more data VIA RPC, moved some code back into the BN only - attestation duties being requested as well
2020-05-27 17:06:28 +00:00
var res = -1
for i, it {.inject.} in s:
if predicate:
res = i
break
res