status-im
/
nimbus-eth2
mirror of https://github.com/status-im/nimbus-eth2.git
2
0
Fork 0
Code Issues Projects Releases Wiki Activity
nimbus-eth2/beacon_chain/beacon_node_common.nim

89 lines
2.7 KiB
Nim
Raw Normal View History

Separate validator duties from the Beacon Node (#976) * Separate validator duties from the Beacon Node * remove duplicate MaxEmptySlotCOunt * imports * have beacnde properly compile
2020-05-06 13:23:45 +00:00
# beacon_chain
partially integrate eth1 merge changes (#2548) * partially integrate eth1 merge changes * use hexToSeqByte() and validate execution engine opaque transaction length * remove incorrect REST serialization code
2021-05-20 10:44:13 +00:00
# Copyright (c) 2018-2021 Status Research & Development GmbH
Separate validator duties from the Beacon Node (#976) * Separate validator duties from the Beacon Node * remove duplicate MaxEmptySlotCOunt * imports * have beacnde properly compile
2020-05-06 13:23:45 +00:00
# Licensed and distributed under either of
# * MIT license (license terms in the root directory or at https://opensource.org/licenses/MIT).
# * Apache v2 license (license terms in the root directory or at https://www.apache.org/licenses/LICENSE-2.0).
# at your option. This file may not be copied, modified, or distributed except according to those terms.
# Common routines for a BeaconNode and a BeaconValidator node
import
handle fork choice selecting a head unknown to chaindag (#1907) * move head slot metric to head selection * avoid redundant head selection in validator duties
2020-10-28 07:55:36 +00:00
std/osproc,
first batch of work towards the VC/BN split: - we have a new binary which connects via RPC to the respective BN and has an internal clock - waking it up on every slot - the BN has a new option called --external-validators and currently in order to have the VC binaries to run we need to pass EXTERNAL_VALIDATORS=yes to make - factored some code out of beacon_node.nim for easier reuse in validator_api.nim and validator_client.nim - the VC loads its associated private keys from the datadir for its BN - most of the validator API calls have been implemented as a stub. - the VC polls its BN at the start of each epoch - getting a list of all active validators for the current epoch - and then continues to request blocks and sign them with its appropriate validators when necessary
2020-05-22 17:04:52 +00:00
Separate validator duties from the Beacon Node (#976) * Separate validator duties from the Beacon Node * remove duplicate MaxEmptySlotCOunt * imports * have beacnde properly compile
2020-05-06 13:23:45 +00:00
# Nimble packages
Add a POST /api/nimbus/v1/graffiti API
2021-03-26 17:57:02 +00:00
chronos, json_rpc/servers/httpserver, presto,
Parallel attestation verification (#2718) * Add parallel attestation verification * Update tests, batchVerify doesn't use the threadpool with only single core (nim-blscurve update) * bump nim-blscurve * Debug info for failing eth2 test vectors * remove submodule eth2-testnets * verbose debugging of make failure on Windows (libbacktrace?) * Remove CI debug mode * initialization convention * Fix new altair tests
2021-09-17 00:13:52 +00:00
taskpools,
Separate validator duties from the Beacon Node (#976) * Separate validator duties from the Beacon Node * remove duplicate MaxEmptySlotCOunt * imports * have beacnde properly compile
2020-05-06 13:23:45 +00:00
# Local modules
Reorg (5/5) (#2377) * Reorg things left into networking and gossip_processing * time -> beacon_clock * fix builds
2021-03-05 13:12:00 +00:00
./conf, ./beacon_clock, ./beacon_chain_db,
Move SSZ, deposit_contracts & eth1_monitor [reorg files 3/5] (#2371) * move deposit_contract * Move SSZ * fix ssz import in tests * move also eth1_monitor * forgot to delete the original * fix comma [skip ci] * Fix "make" & tools imports * Fix import * Fix import again * rename deposit_contract -> eth1 * Revert ssz move to subfolder * path fixes [skip ci]
2021-03-03 06:23:05 +00:00
./beacon_node_types,
clean up block processing (#2610) * gossip_to_consensus -> block_processor (it's processing only blocks, but not only from gossip) * measure queue and validation time for blocks * measure assignment and state loading times for updateStateData * avoid some unnecessary block copies in block sync * warn that database is corrupt if we hit tail without a state
2021-05-28 16:34:00 +00:00
./gossip_processing/[eth2_processor, block_processor, consensus_manager],
Reorg (5/5) (#2377) * Reorg things left into networking and gossip_processing * time -> beacon_clock * fix builds
2021-03-05 13:12:00 +00:00
./networking/eth2_network,
Move SSZ, deposit_contracts & eth1_monitor [reorg files 3/5] (#2371) * move deposit_contract * Move SSZ * fix ssz import in tests * move also eth1_monitor * forgot to delete the original * fix comma [skip ci] * Fix "make" & tools imports * Fix import * Fix import again * rename deposit_contract -> eth1 * Revert ssz move to subfolder * path fixes [skip ci]
2021-03-03 06:23:05 +00:00
./eth1/eth1_monitor,
Consensus object pools [reorg 4/5] (#2374) * Add documentation * make test doesn't try to build the beacon node :/
2021-03-04 09:13:44 +00:00
./consensus_object_pools/[blockchain_dag, block_quarantine, attestation_pool],
remove false OnBlockAdded dependency on phase0 HashedBeaconState (#2661) * remove false OnBlockAdded dependency on phase.HashedBeaconState * introduce altair data types into block_clearance; update some alpha.6 spec refs to alpha.7; add get_active_validator_indices_len ForkedHashedBeaconState wrapper * switch many modules from using datatypes (with phase0 states/blocks) to datatypes/base (fork-independent); update spec refs from alpha.6 to alpha.7 and remove rm'd G2_POINT_AT_INFINITY * switch more modules from using datatypes (with phase0 states/blocks) to datatypes/base (fork-independent); update spec refs from alpha.6 to alpha.7 * remove unnecessary phase0-only wrapper of get_attesting_indices(); allow signatures_batch to process either fork; remove O(n^2) nested loop in process_inactivity_updates(); add altair support to getAttestationsforTestBlock() * add Altair versions of asSigVerified(), asTrusted(), and makeBeaconBlock() * fix spec URL to be Altair for Altair makeBeaconBlock()
2021-06-21 08:35:24 +00:00
./spec/datatypes/base,
Consolidate modules by areas [part 1] (#2365) * Move sync in subfolder * move validator related thingies in validators * fix binary builds * update bounds comment [skip ci]
2021-03-02 10:27:45 +00:00
./sync/[sync_manager, request_manager]
Separate validator duties from the Beacon Node (#976) * Separate validator duties from the Beacon Node * remove duplicate MaxEmptySlotCOunt * imports * have beacnde properly compile
2020-05-06 13:23:45 +00:00
handle fork choice selecting a head unknown to chaindag (#1907) * move head slot metric to head selection * avoid redundant head selection in validator duties
2020-10-28 07:55:36 +00:00
export
REST API client, JSON-RPC cleanups (#2756) This refactoring puts the JSON-RPC and REST APIs on more equal footing by renaming and moving things around, creating a separation between client and server, and documenting what they are - the aim is to have a simple-to-use base to start from when developing API clients, as well as make it easier to navigate the code when looking for the legacy JSON-RPC interface vs the new REST API. * move REST client, serialization and supporting types to spec/eth2_apis * REST stuff now starts with `rest_`, JSON-RPC stuff starts with `rpc_`, more or less * simplify imports such that there's a simple module to import for both server and client * map REST type and proc names to yaml spec more closely - in particular, reuse operation and type names in `rest_types` to make comparisons against spec more easy * cleaner separation between client and server modules - modules common between server and client such as `rest_types` and serialization move to the spec folder - this allows the client to be built with less knowledge about server internals
2021-08-03 15:17:11 +00:00
osproc, chronos, httpserver, presto, conf, beacon_clock, beacon_chain_db,
Rename mainchain_monitor to eth1_monitor
2020-11-03 19:02:43 +00:00
attestation_pool, eth2_network, beacon_node_types, eth1_monitor,
Consensus object pools [reorg 4/5] (#2374) * Add documentation * make test doesn't try to build the beacon node :/
2021-03-04 09:13:44 +00:00
request_manager, sync_manager, eth2_processor, blockchain_dag, block_quarantine,
remove false OnBlockAdded dependency on phase0 HashedBeaconState (#2661) * remove false OnBlockAdded dependency on phase.HashedBeaconState * introduce altair data types into block_clearance; update some alpha.6 spec refs to alpha.7; add get_active_validator_indices_len ForkedHashedBeaconState wrapper * switch many modules from using datatypes (with phase0 states/blocks) to datatypes/base (fork-independent); update spec refs from alpha.6 to alpha.7 and remove rm'd G2_POINT_AT_INFINITY * switch more modules from using datatypes (with phase0 states/blocks) to datatypes/base (fork-independent); update spec refs from alpha.6 to alpha.7 * remove unnecessary phase0-only wrapper of get_attesting_indices(); allow signatures_batch to process either fork; remove O(n^2) nested loop in process_inactivity_updates(); add altair support to getAttestationsforTestBlock() * add Altair versions of asSigVerified(), asTrusted(), and makeBeaconBlock() * fix spec URL to be Altair for Altair makeBeaconBlock()
2021-06-21 08:35:24 +00:00
base
first batch of work towards the VC/BN split: - we have a new binary which connects via RPC to the respective BN and has an internal clock - waking it up on every slot - the BN has a new option called --external-validators and currently in order to have the VC binaries to run we need to pass EXTERNAL_VALIDATORS=yes to make - factored some code out of beacon_node.nim for easier reuse in validator_api.nim and validator_client.nim - the VC loads its associated private keys from the datadir for its BN - most of the validator API calls have been implemented as a stub. - the VC polls its BN at the start of each epoch - getting a list of all active validators for the current epoch - and then continues to request blocks and sign them with its appropriate validators when necessary
2020-05-22 17:04:52 +00:00
Separate validator duties from the Beacon Node (#976) * Separate validator duties from the Beacon Node * remove duplicate MaxEmptySlotCOunt * imports * have beacnde properly compile
2020-05-06 13:23:45 +00:00
type
RpcServer* = RpcHttpServer
Parallel attestation verification (#2718) * Add parallel attestation verification * Update tests, batchVerify doesn't use the threadpool with only single core (nim-blscurve update) * bump nim-blscurve * Debug info for failing eth2 test vectors * remove submodule eth2-testnets * verbose debugging of make failure on Windows (libbacktrace?) * Remove CI debug mode * initialization convention * Fix new altair tests
2021-09-17 00:13:52 +00:00
TaskPoolPtr* = TaskPool
Separate validator duties from the Beacon Node (#976) * Separate validator duties from the Beacon Node * remove duplicate MaxEmptySlotCOunt * imports * have beacnde properly compile
2020-05-06 13:23:45 +00:00
altair topic switching (#2767) * altair topic switching * remove validate{Committee,Validator}IndexOr unused within branch
2021-08-09 12:54:45 +00:00
GossipState* = enum
Disconnected
ConnectedToPhase0
InTransitionToAltair
ConnectedToAltair
Separate validator duties from the Beacon Node (#976) * Separate validator duties from the Beacon Node * remove duplicate MaxEmptySlotCOunt * imports * have beacnde properly compile
2020-05-06 13:23:45 +00:00
BeaconNode* = ref object
nickname*: string
Add an option for graffiti customization
2020-06-29 17:30:19 +00:00
graffitiBytes*: GraffitiBytes
Separate validator duties from the Beacon Node (#976) * Separate validator duties from the Beacon Node * remove duplicate MaxEmptySlotCOunt * imports * have beacnde properly compile
2020-05-06 13:23:45 +00:00
network*: Eth2Node
Bump modules to take advantage of the new Json format flavors support Since quite a lot of additional procs were now compiled as generics, this lead to compiler bugs that had to be worked-around: * The `Domain` type was renamed to `Eth2Domain` to avoid compilation errors due to conflicts with `nativesockets.Domain`. Similarly, `eth2_network.KeyPair` was renamed to `NetKeyPair`. * A new more robust version of `hexToByteArray` was added to stew
2021-03-19 02:22:45 +00:00
netKeys*: NetKeyPair
Separate validator duties from the Beacon Node (#976) * Separate validator duties from the Beacon Node * remove duplicate MaxEmptySlotCOunt * imports * have beacnde properly compile
2020-05-06 13:23:45 +00:00
db*: BeaconChainDB
config*: BeaconNodeConf
startup cleanup * fix several memory leaks due to temporaries not being reset during init * avoid massive main() function with lots of stuff in it * disable nim-prompt (unused) * reuse validator pool instance in eth2_processor * style cleanup
2021-02-22 16:17:48 +00:00
attachedValidators*: ref ValidatorPool
singe validator key cache Instead of keeping a validator key list per EpochRef, this PR introduces a single shared validator key list in ChainDAG, and cleans up some other ChainDAG and key-related issues. The PR does not introduce the validator key list in the state transition - this is because we batch-check all signatures before entering the spec code, thus the spec code never hits the cache. A future refactor should _probably_ remove the threadvar altogether. There's a few other small fixes in here that make the flow easier to read: * fix `var ChainDAGRef` -> `ChainDAGRef` * fix `var QuarantineRef` -> `QuarantineRef` * consistent `dag` variable name * avoid using threadvar pubkey cache in most cases * better error messages in batch signature checking
2021-06-01 11:13:40 +00:00
dag*: ChainDAGRef
renamed CandidateChains to ChainDagRef and made the Quarantine type a ref type so there is a single instance in the beacon node (#1407)
2020-07-31 14:49:06 +00:00
quarantine*: QuarantineRef
avoid double deserialization When blocks and attestations arrive, they are SSZ-decoded twice: once for validation and once for processing. This branch enqueues the decoded block directly for processing, avoiding the second, slow deserialization. * move processing of blocks and attestations to queue * ...and out from beacon_node * split attestation processing into attestations and aggregates * also updates metrics * clean up logging to better follow the lifetime of gossip: arrival, validation and processing * drop attestations and aggregates if there are too many * try to prioritise blocks and aggregates before single-validator attestations
2020-08-20 16:30:47 +00:00
attestationPool*: ref AttestationPool
introduce SyncCommitteeMsgPool to eth2_processor and nimbus_beacon_node (#2831)
2021-08-28 22:27:51 +00:00
syncCommitteeMsgPool*: ref SyncCommitteeMsgPool
[WIP] skeleton of attester slashing pool & validators (#1639) * skeleton of attester slashing pool & validators * add skeleton for proposer slashings and voluntary exits; rename pool to more inclusive exit pool to stay consistent with all three; ensure is initialized by beacon_node so is safe to merge, even if it doesn't do much yet
2020-09-14 14:26:31 +00:00
exitPool*: ref ExitPool
Further reduce the number of RPC requests; Significantly faster ETH1 syncing
2020-11-03 01:21:07 +00:00
eth1Monitor*: Eth1Monitor
Separate validator duties from the Beacon Node (#976) * Separate validator duties from the Beacon Node * remove duplicate MaxEmptySlotCOunt * imports * have beacnde properly compile
2020-05-06 13:23:45 +00:00
rpcServer*: RpcServer
Rest API initial implementation.
2021-03-17 18:46:45 +00:00
restServer*: RestServerRef
REST /eth/v1/events API call implementation. (#2878) * Placing callbacks into strategic places. * Initial events call implementation. * Post rebase fixes. * Change addSyncContribution() implementation. * Add `attestation-sent` event. Remove gcsafe, raises from callbacks implementations. Move `attestation-received` fire at the end of attestation processing. * Address review comments.
2021-09-22 12:17:15 +00:00
eventBus*: AsyncEventBus
use a separate process for the private keys (Off by default) - there is a new signing_process binary which loads all validators of the beacon node and the BN dictates through stdin of the signing process what to be signed and when and reads from stdout of the process
2020-09-01 13:44:40 +00:00
vcProcess*: Process
Use one single async queue and loop for processing blocks. (#1487) * Initial commit * Fix compilation problem. * Address review comments.
2020-08-12 09:29:11 +00:00
requestManager*: RequestManager
Add proper concurrent connections. Add SeenTable to avoid continuous attempts to dead peers. Refactor onSecond. Block backward sync while forward sync is working. SyncManager now checks responses according corresponding requests + tests. SyncManager now watching for not progressing local_head_slot and resets SyncQueue.
2020-06-03 08:46:29 +00:00
syncManager*: SyncManager[Peer, PeerID]
Embed the Altona metadata in the NBC executable
2020-07-02 15:14:11 +00:00
genesisSnapshotContent*: string
initial dynamic subscribe/unsubscribe for attestations to/from subnets (#1462) * initial dynamic subscribe/unsubscribe for attestations to/from subnets * implement random stability subnet and clean up * switch from HashSet[uint64] to set[uint8] * refactor subnet logic out from beacon_node and actual (un)subscribing * only try to subscribe to marginally different subnets * add assertions * maintain ENR subnets * assert that beacon_node and eth2_network have consistent view of subscribed subnets * disable actual cycling
2020-08-12 17:48:31 +00:00
attestationSubnets*: AttestationSubnets
avoid double deserialization When blocks and attestations arrive, they are SSZ-decoded twice: once for validation and once for processing. This branch enqueues the decoded block directly for processing, avoiding the second, slow deserialization. * move processing of blocks and attestations to queue * ...and out from beacon_node * split attestation processing into attestations and aggregates * also updates metrics * clean up logging to better follow the lifetime of gossip: arrival, validation and processing * drop attestations and aggregates if there are too many * try to prioritise blocks and aggregates before single-validator attestations
2020-08-20 16:30:47 +00:00
processor*: ref Eth2Processor
clean up block processing (#2610) * gossip_to_consensus -> block_processor (it's processing only blocks, but not only from gossip) * measure queue and validation time for blocks * measure assignment and state loading times for updateStateData * avoid some unnecessary block copies in block sync * warn that database is corrupt if we hit tail without a state
2021-05-28 16:34:00 +00:00
blockProcessor*: ref BlockProcessor
Split Eth2Processor in prep for batching (#2396) * Split Eth2Processor in gossip and consensus part and materialize the shared block queue * Update initialization in test_sync_manager
2021-03-11 10:10:57 +00:00
consensusManager*: ref ConsensusManager
Don't rely on a metric value for the ETH display in the status bar
2020-11-29 21:35:39 +00:00
attachedValidatorBalanceTotal*: uint64
altair topic switching (#2767) * altair topic switching * remove validate{Committee,Validator}IndexOr unused within branch
2021-08-09 12:54:45 +00:00
gossipState*: GossipState
move BeaconClock from ChainDAG to BeaconNode (#2796)
2021-08-20 08:58:15 +00:00
beaconClock*: BeaconClock
Parallel attestation verification (#2718) * Add parallel attestation verification * Update tests, batchVerify doesn't use the threadpool with only single core (nim-blscurve update) * bump nim-blscurve * Debug info for failing eth2 test vectors * remove submodule eth2-testnets * verbose debugging of make failure on Windows (libbacktrace?) * Remove CI debug mode * initialization convention * Fix new altair tests
2021-09-17 00:13:52 +00:00
taskpool*: TaskPoolPtr
REST /eth/v1/events API call implementation. (#2878) * Placing callbacks into strategic places. * Initial events call implementation. * Post rebase fixes. * Change addSyncContribution() implementation. * Add `attestation-sent` event. Remove gcsafe, raises from callbacks implementations. Move `attestation-received` fire at the end of attestation processing. * Address review comments.
2021-09-22 12:17:15 +00:00
onAttestationSent*: OnAttestationCallback
Number of REST fixes. (#2987) * Initial commit. * Fix path. * Add validator keys to indices cache mechanism. Move syncComitteeParticipants to common place. * Fix sync participants order issue. * Fix error code when state could not be found. Refactor `state/validators` to use keysToIndices mechanism. * Fix RestValidatorIndex to ValidatorIndex conversion TODOs. * Address review comments. * Fix REST test rules.
2021-10-14 10:38:38 +00:00
restKeysCache*: Table[ValidatorPubKey, ValidatorIndex]
Separate validator duties from the Beacon Node (#976) * Separate validator duties from the Beacon Node * remove duplicate MaxEmptySlotCOunt * imports * have beacnde properly compile
2020-05-06 13:23:45 +00:00
first batch of work towards the VC/BN split: - we have a new binary which connects via RPC to the respective BN and has an internal clock - waking it up on every slot - the BN has a new option called --external-validators and currently in order to have the VC binaries to run we need to pass EXTERNAL_VALIDATORS=yes to make - factored some code out of beacon_node.nim for easier reuse in validator_api.nim and validator_client.nim - the VC loads its associated private keys from the datadir for its BN - most of the validator API calls have been implemented as a stub. - the VC polls its BN at the start of each epoch - getting a list of all active validators for the current epoch - and then continues to request blocks and sign them with its appropriate validators when necessary
2020-05-22 17:04:52 +00:00
const
MaxEmptySlotCount* = uint64(10*60) div SECONDS_PER_SLOT
Separate validator duties from the Beacon Node (#976) * Separate validator duties from the Beacon Node * remove duplicate MaxEmptySlotCOunt * imports * have beacnde properly compile
2020-05-06 13:23:45 +00:00
stop discarding future epochs; remove a StateCache() construction (#1610) * stop discarding non-existent future epochs during epoch state transitions; remove a pointless StateCache() construction in advance_slots() * update nbench to pass StateCache to process_slots()
2020-09-07 15:04:33 +00:00
# TODO stew/sequtils2
openarray -> openArray
2020-10-28 18:35:31 +00:00
template findIt*(s: openArray, predicate: untyped): int =
more work on the BN/VC split - fixed comments from the last review - getting more data VIA RPC, moved some code back into the BN only - attestation duties being requested as well
2020-05-27 17:06:28 +00:00
var res = -1
for i, it {.inject.} in s:
if predicate:
res = i
break
res
move rpc api to own folder, mimic upstream structure (#1905) also implements a few more endpoints
2020-10-27 09:00:57 +00:00
proc currentSlot*(node: BeaconNode): Slot =
node.beaconClock.now.slotOrZero