status-im
/
nimbus-eth2
mirror of https://github.com/status-im/nimbus-eth2.git
2
0
Fork 0
Code Issues Projects Releases Wiki Activity
nimbus-eth2/beacon_chain/el/el_manager.nim

2255 lines
86 KiB
Nim
Raw Normal View History

EH cleanup (#2455) almost 100% raises in nimbus-eth2 now! * fix some rare exception-related crashes in json-rpc
2021-03-26 06:52:01 +00:00
# beacon_chain
capella forkchoiceUpdated support (#4462) * capella forkchoiceUpdated support * match V2 fcU with V2 getPayload
2023-01-06 21:01:10 +00:00
# Copyright (c) 2018-2023 Status Research & Development GmbH
EH cleanup (#2455) almost 100% raises in nimbus-eth2 now! * fix some rare exception-related crashes in json-rpc
2021-03-26 06:52:01 +00:00
# Licensed and distributed under either of
# * MIT license (license terms in the root directory or at https://opensource.org/licenses/MIT).
# * Apache v2 license (license terms in the root directory or at https://www.apache.org/licenses/LICENSE-2.0).
# at your option. This file may not be copied, modified, or distributed except according to those terms.
remove Nim 1.2-compatible `push raise`s and update copyright notice years (#4528)
2023-01-20 14:14:37 +00:00
{.push raises: [].}
EH cleanup (#2455) almost 100% raises in nimbus-eth2 now! * fix some rare exception-related crashes in json-rpc
2021-03-26 06:52:01 +00:00
Add more flesh to the skeleton :)
2018-11-26 13:33:06 +00:00
import
remove most std/options imports (#4778)
2023-03-31 20:46:47 +00:00
std/[deques, strformat, strutils, sequtils, tables, typetraits, uri, json],
Add some useful Eth1 monitor metrics
2020-12-09 22:44:59 +00:00
# Nimble packages:
Bump stint to v2.0: new array backend (#5113) * bump stint to v2.0: new array backend * Fix missing isEven in libnimbus_lc.nim * bump nim-stint: compiles with arm64
2023-09-12 18:16:04 +00:00
chronos, metrics, chronicles/timings,
Support for driving multiple EL nodes from a single Nimbus BN (#4465) * Support for driving multiple EL nodes from a single Nimbus BN Full list of changes: * Eth1Monitor has been renamed to ELManager to match its current responsibilities better. * The ELManager is no longer optional in the code (it won't have a nil value under any circumstances). * The support for subscribing for headers was removed as it only worked with WebSockets and contributed significant complexity while bringing only a very minor advantage. * The `--web3-url` parameter has been deprecated in favor of a new `--el` parameter. The new parameter has a reasonable default value and supports specifying a different JWT for each connection. Each connection can also be configured with a different set of responsibilities (e.g. download deposits, validate blocks and/or produce blocks). On the command-line, these properties can be configured through URL properties stored in the #anchor part of the URL. In TOML files, they come with a very natural syntax (althrough the URL scheme is also supported). * The previously scattered EL-related state and logic is now moved to `eth1_monitor.nim` (this module will be renamed to `el_manager.nim` in a follow-up commit). State is assigned properly either to the `ELManager` or the to individual `ELConnection` objects where appropriate. The ELManager executes all Engine API requests against all attached EL nodes, in parallel. It compares their results and if there is a disagreement regarding the validity of a certain payload, this is detected and the beacon node is protected from publishing a block with a potential execution layer consensus bug in it. The BN provides metrics per EL node for the number of successful or failed requests for each type Engine API requests. If an EL node goes offline and connectivity is resoted later, we report the problem and the remedy in edge-triggered fashion. * More progress towards implementing Deneb block production in the VC and comparing the value of blocks produced by the EL and the builder API. * Adds a Makefile target for the zhejiang testnet
2023-03-05 01:40:21 +00:00
json_rpc/[client, errors],
web3, web3/ethhexstrings, web3/engine_api,
eth/common/[eth_types, transaction],
Bump stint to v2.0: new array backend (#5113) * bump stint to v2.0: new array backend * Fix missing isEven in libnimbus_lc.nim * bump nim-stint: compiles with arm64
2023-09-12 18:16:04 +00:00
eth/async_utils, stew/[assign2, byteutils, objects, results, shims/hashes, endians2],
Add some useful Eth1 monitor metrics
2020-12-09 22:44:59 +00:00
# Local modules:
Support for obtaining deposit snapshots during trustedNodeSync (#4303) Other changes: * More optimal search for TTD block. * Add timeouts to all REST requests during trusted node sync. Fixes #4037 * Removed support for storing a deposit snapshot in the network metadata.
2022-12-07 10:24:51 +00:00
../spec/[deposit_snapshots, eth2_merkleization, forks, helpers],
more eip4844 -> deneb changes (#4666)
2023-02-25 01:03:34 +00:00
../spec/datatypes/[base, phase0, bellatrix, deneb],
Reorg (5/5) (#2377) * Reorg things left into networking and gossip_processing * time -> beacon_clock * fix builds
2021-03-05 13:12:00 +00:00
../networking/network_metadata,
use StateData in place of BeaconState outside state transition code (#2551) * use StateData in place of BeaconState outside state transition code * propagate more StateData usage * remove withStateVars().state * wrap get_beacon_committee(BeaconState, ...) as gbc(StateData, ...) * switch makeAttestation() to use StateData * use StateData wrapper/dispatcher for get_committee_count_per_slot() * convert AttestationCache.init(), weak subjectivity functions, and updateValidatorMetrics() * add get_shuffled_active_validator_indices(StateData) and get_block_root_at_slot(StateData) * switch makeAttestationData() to StateData * sync AllTests-mainnet.md after rebase
2021-05-21 09:23:28 +00:00
../consensus_object_pools/block_pools_types,
Support for driving multiple EL nodes from a single Nimbus BN (#4465) * Support for driving multiple EL nodes from a single Nimbus BN Full list of changes: * Eth1Monitor has been renamed to ELManager to match its current responsibilities better. * The ELManager is no longer optional in the code (it won't have a nil value under any circumstances). * The support for subscribing for headers was removed as it only worked with WebSockets and contributed significant complexity while bringing only a very minor advantage. * The `--web3-url` parameter has been deprecated in favor of a new `--el` parameter. The new parameter has a reasonable default value and supports specifying a different JWT for each connection. Each connection can also be configured with a different set of responsibilities (e.g. download deposits, validate blocks and/or produce blocks). On the command-line, these properties can be configured through URL properties stored in the #anchor part of the URL. In TOML files, they come with a very natural syntax (althrough the URL scheme is also supported). * The previously scattered EL-related state and logic is now moved to `eth1_monitor.nim` (this module will be renamed to `el_manager.nim` in a follow-up commit). State is assigned properly either to the `ELManager` or the to individual `ELConnection` objects where appropriate. The ELManager executes all Engine API requests against all attached EL nodes, in parallel. It compares their results and if there is a disagreement regarding the validity of a certain payload, this is detected and the beacon node is protected from publishing a block with a potential execution layer consensus bug in it. The BN provides metrics per EL node for the number of successful or failed requests for each type Engine API requests. If an EL node goes offline and connectivity is resoted later, we report the problem and the remedy in edge-triggered fashion. * More progress towards implementing Deneb block production in the VC and comparing the value of blocks produced by the EL and the builder API. * Adds a Makefile target for the zhejiang testnet
2023-03-05 01:40:21 +00:00
".."/[beacon_chain_db, beacon_node_status, beacon_clock, future_combinators],
"."/[merkle_minimal, el_conf]
Add more flesh to the skeleton :)
2018-11-26 13:33:06 +00:00
JWT support (#3561)
2022-03-31 14:43:05 +00:00
from std/times import getTime, inSeconds, initTime, `-`
from ../spec/engine_authentication import getSignedIatToken
https://github.com/ethereum/consensus-specs/pull/3359 (#5047)
2023-06-10 05:39:10 +00:00
from ../spec/state_transition_block import kzg_commitment_to_versioned_hash
JWT support (#3561)
2022-03-31 14:43:05 +00:00
Smooth out the differences between Ganache and Infura; Working local sim and Altona target
2020-06-19 17:42:28 +00:00
export
Support for driving multiple EL nodes from a single Nimbus BN (#4465) * Support for driving multiple EL nodes from a single Nimbus BN Full list of changes: * Eth1Monitor has been renamed to ELManager to match its current responsibilities better. * The ELManager is no longer optional in the code (it won't have a nil value under any circumstances). * The support for subscribing for headers was removed as it only worked with WebSockets and contributed significant complexity while bringing only a very minor advantage. * The `--web3-url` parameter has been deprecated in favor of a new `--el` parameter. The new parameter has a reasonable default value and supports specifying a different JWT for each connection. Each connection can also be configured with a different set of responsibilities (e.g. download deposits, validate blocks and/or produce blocks). On the command-line, these properties can be configured through URL properties stored in the #anchor part of the URL. In TOML files, they come with a very natural syntax (althrough the URL scheme is also supported). * The previously scattered EL-related state and logic is now moved to `eth1_monitor.nim` (this module will be renamed to `el_manager.nim` in a follow-up commit). State is assigned properly either to the `ELManager` or the to individual `ELConnection` objects where appropriate. The ELManager executes all Engine API requests against all attached EL nodes, in parallel. It compares their results and if there is a disagreement regarding the validity of a certain payload, this is detected and the beacon node is protected from publishing a block with a potential execution layer consensus bug in it. The BN provides metrics per EL node for the number of successful or failed requests for each type Engine API requests. If an EL node goes offline and connectivity is resoted later, we report the problem and the remedy in edge-triggered fashion. * More progress towards implementing Deneb block production in the VC and comparing the value of blocks produced by the EL and the builder API. * Adds a Makefile target for the zhejiang testnet
2023-03-05 01:40:21 +00:00
el_conf, engine_api, deques, base, DepositTreeSnapshot
Smooth out the differences between Ganache and Infura; Working local sim and Altona target
2020-06-19 17:42:28 +00:00
Tentative fix for the Pyrmont Eth1 voting issue
2020-11-19 17:19:03 +00:00
logScope:
Support for driving multiple EL nodes from a single Nimbus BN (#4465) * Support for driving multiple EL nodes from a single Nimbus BN Full list of changes: * Eth1Monitor has been renamed to ELManager to match its current responsibilities better. * The ELManager is no longer optional in the code (it won't have a nil value under any circumstances). * The support for subscribing for headers was removed as it only worked with WebSockets and contributed significant complexity while bringing only a very minor advantage. * The `--web3-url` parameter has been deprecated in favor of a new `--el` parameter. The new parameter has a reasonable default value and supports specifying a different JWT for each connection. Each connection can also be configured with a different set of responsibilities (e.g. download deposits, validate blocks and/or produce blocks). On the command-line, these properties can be configured through URL properties stored in the #anchor part of the URL. In TOML files, they come with a very natural syntax (althrough the URL scheme is also supported). * The previously scattered EL-related state and logic is now moved to `eth1_monitor.nim` (this module will be renamed to `el_manager.nim` in a follow-up commit). State is assigned properly either to the `ELManager` or the to individual `ELConnection` objects where appropriate. The ELManager executes all Engine API requests against all attached EL nodes, in parallel. It compares their results and if there is a disagreement regarding the validity of a certain payload, this is detected and the beacon node is protected from publishing a block with a potential execution layer consensus bug in it. The BN provides metrics per EL node for the number of successful or failed requests for each type Engine API requests. If an EL node goes offline and connectivity is resoted later, we report the problem and the remedy in edge-triggered fashion. * More progress towards implementing Deneb block production in the VC and comparing the value of blocks produced by the EL and the builder API. * Adds a Makefile target for the zhejiang testnet
2023-03-05 01:40:21 +00:00
topics = "elmon"
Tentative fix for the Pyrmont Eth1 voting issue
2020-11-19 17:19:03 +00:00
The deposit contract uses dynamic bytes for most of its event properties https://github.com/ethereum/consensus-specs/blob/dev/solidity_deposit_contract/deposit_contract.sol#L19-L24
2021-10-14 12:33:55 +00:00
type
PubKeyBytes = DynamicBytes[48, 48]
WithdrawalCredentialsBytes = DynamicBytes[32, 32]
SignatureBytes = DynamicBytes[96, 96]
Int64LeBytes = DynamicBytes[8, 8]
Spec-compliant implementation of Eth1 monitoring; Eth1-enabled local sim BEWARE! This commit will trigger a stack overflow during local sim
2020-03-24 11:13:07 +00:00
contract(DepositContract):
The deposit contract uses dynamic bytes for most of its event properties https://github.com/ethereum/consensus-specs/blob/dev/solidity_deposit_contract/deposit_contract.sol#L19-L24
2021-10-14 12:33:55 +00:00
proc deposit(pubkey: PubKeyBytes,
withdrawalCredentials: WithdrawalCredentialsBytes,
signature: SignatureBytes,
Spec-compliant implementation of Eth1 monitoring; Eth1-enabled local sim BEWARE! This commit will trigger a stack overflow during local sim
2020-03-24 11:13:07 +00:00
deposit_data_root: FixedBytes[32])
proc get_deposit_root(): FixedBytes[32]
The deposit contract uses dynamic bytes for most of its event properties https://github.com/ethereum/consensus-specs/blob/dev/solidity_deposit_contract/deposit_contract.sol#L19-L24
2021-10-14 12:33:55 +00:00
proc get_deposit_count(): Int64LeBytes
Spec-compliant implementation of Eth1 monitoring; Eth1-enabled local sim BEWARE! This commit will trigger a stack overflow during local sim
2020-03-24 11:13:07 +00:00
The deposit contract uses dynamic bytes for most of its event properties https://github.com/ethereum/consensus-specs/blob/dev/solidity_deposit_contract/deposit_contract.sol#L19-L24
2021-10-14 12:33:55 +00:00
proc DepositEvent(pubkey: PubKeyBytes,
withdrawalCredentials: WithdrawalCredentialsBytes,
amount: Int64LeBytes,
signature: SignatureBytes,
index: Int64LeBytes) {.event.}
Spec-compliant implementation of Eth1 monitoring; Eth1-enabled local sim BEWARE! This commit will trigger a stack overflow during local sim
2020-03-24 11:13:07 +00:00
Fix the corrupted database state on Pyrmont nodes; Add mainnet genesis (#2056) * Handle some web3 timeouts better * Add support for developer .env files * Eth1 improvements; Mainnet genesis state Notable changes: * The deposits table have been removed from the database. The client will no longer process all deposits on start-up. * The network metadata now includes a "state snapshot" of the deposit contract. This allows the client to skip syncing deposits made prior to the snapshot (i.e. genesis). Suitable metadata added for Pyrmont and Mainnet. * The Eth1 monitor won't be started unless there are validators attached to the node. * The genesis detection code is now optional and disabled by default * Bugfix: The client should not produce blocks that will fail validation when it hasn't downloaded the latest deposits yet * Bugfix: Work around the database corruption affecting Pyrmont nodes * Remove metadata for Toledo and Medalla
2020-11-24 21:21:47 +00:00
const
Remove some accidentally left debugging changes
2020-12-15 23:09:19 +00:00
hasDepositRootChecks = defined(has_deposit_root_checks)
Fix the corrupted database state on Pyrmont nodes; Add mainnet genesis (#2056) * Handle some web3 timeouts better * Add support for developer .env files * Eth1 improvements; Mainnet genesis state Notable changes: * The deposits table have been removed from the database. The client will no longer process all deposits on start-up. * The network metadata now includes a "state snapshot" of the deposit contract. This allows the client to skip syncing deposits made prior to the snapshot (i.e. genesis). Suitable metadata added for Pyrmont and Mainnet. * The Eth1 monitor won't be started unless there are validators attached to the node. * The genesis detection code is now optional and disabled by default * Bugfix: The client should not produce blocks that will fail validation when it hasn't downloaded the latest deposits yet * Bugfix: Work around the database corruption affecting Pyrmont nodes * Remove metadata for Toledo and Medalla
2020-11-24 21:21:47 +00:00
Lower the maximum block range in getLogs requests to 1000
2023-03-14 11:45:01 +00:00
targetBlocksPerLogsRequest = 1000'u64
# TODO
#
# This is currently set to 1000, because this was the default maximum
# value in Besu circa our 22.3.0 release. Previously, we've used 5000,
# but this was effectively forcing the fallback logic in `syncBlockRange`
# to always execute multiple requests before getting a successful response.
#
# Besu have raised this default to 5000 in https://github.com/hyperledger/besu/pull/5209
# which is expected to ship in their next release.
#
# Full deposits sync time with various values for this parameter:
#
# Blocks per request | Geth running on the same host | Geth running on a more distant host
# ----------------------------------------------------------------------------------------
# 1000 | 11m 20s | 22m
# 5000 | 5m 20s | 15m 40s
# 100000 | 4m 10s | not tested
#
# The number of requests scales linearly with the parameter value as you would expect.
#
# These results suggest that it would be reasonable for us to get back to 5000 once the
# Besu release is well-spread within their userbase.
track request chunk size across EL reconnects (#3960) When the EL connection is interrupted, deposits are once more requested in chunks of 5000 blocks. This is a problem when the response takes over a minute to produce and consistently times out as followup requests with lower chunk sizes may no longer work after a request was canceled, e.g., when using Geth with websockets. By keeping track of `blocksPerRequest` across EL reconnections, it is possible to recover from this by avoiding to continuously repeat the initial request with the full 5000 blocks. Also cleans up one more "retry of retry" instance; `DataProviderTimeout` is a `CatchableError` and already handled by the existing retry logic.
2022-08-12 13:51:33 +00:00
Support for driving multiple EL nodes from a single Nimbus BN (#4465) * Support for driving multiple EL nodes from a single Nimbus BN Full list of changes: * Eth1Monitor has been renamed to ELManager to match its current responsibilities better. * The ELManager is no longer optional in the code (it won't have a nil value under any circumstances). * The support for subscribing for headers was removed as it only worked with WebSockets and contributed significant complexity while bringing only a very minor advantage. * The `--web3-url` parameter has been deprecated in favor of a new `--el` parameter. The new parameter has a reasonable default value and supports specifying a different JWT for each connection. Each connection can also be configured with a different set of responsibilities (e.g. download deposits, validate blocks and/or produce blocks). On the command-line, these properties can be configured through URL properties stored in the #anchor part of the URL. In TOML files, they come with a very natural syntax (althrough the URL scheme is also supported). * The previously scattered EL-related state and logic is now moved to `eth1_monitor.nim` (this module will be renamed to `el_manager.nim` in a follow-up commit). State is assigned properly either to the `ELManager` or the to individual `ELConnection` objects where appropriate. The ELManager executes all Engine API requests against all attached EL nodes, in parallel. It compares their results and if there is a disagreement regarding the validity of a certain payload, this is detected and the beacon node is protected from publishing a block with a potential execution layer consensus bug in it. The BN provides metrics per EL node for the number of successful or failed requests for each type Engine API requests. If an EL node goes offline and connectivity is resoted later, we report the problem and the remedy in edge-triggered fashion. * More progress towards implementing Deneb block production in the VC and comparing the value of blocks produced by the EL and the builder API. * Adds a Makefile target for the zhejiang testnet
2023-03-05 01:40:21 +00:00
# Engine API timeouts
engineApiConnectionTimeout = 5.seconds # How much we wait before giving up connecting to the Engine API
web3RequestsTimeout* = 8.seconds # How much we wait for eth_* requests (e.g. eth_getBlockByHash)
update engine API URLs from v1.0.0-beta.2 to beta.3 (#4828)
2023-04-17 20:11:28 +00:00
# https://github.com/ethereum/execution-apis/blob/v1.0.0-beta.3/src/engine/paris.md#request-2
# https://github.com/ethereum/execution-apis/blob/v1.0.0-beta.3/src/engine/shanghai.md#request-2
Support for driving multiple EL nodes from a single Nimbus BN (#4465) * Support for driving multiple EL nodes from a single Nimbus BN Full list of changes: * Eth1Monitor has been renamed to ELManager to match its current responsibilities better. * The ELManager is no longer optional in the code (it won't have a nil value under any circumstances). * The support for subscribing for headers was removed as it only worked with WebSockets and contributed significant complexity while bringing only a very minor advantage. * The `--web3-url` parameter has been deprecated in favor of a new `--el` parameter. The new parameter has a reasonable default value and supports specifying a different JWT for each connection. Each connection can also be configured with a different set of responsibilities (e.g. download deposits, validate blocks and/or produce blocks). On the command-line, these properties can be configured through URL properties stored in the #anchor part of the URL. In TOML files, they come with a very natural syntax (althrough the URL scheme is also supported). * The previously scattered EL-related state and logic is now moved to `eth1_monitor.nim` (this module will be renamed to `el_manager.nim` in a follow-up commit). State is assigned properly either to the `ELManager` or the to individual `ELConnection` objects where appropriate. The ELManager executes all Engine API requests against all attached EL nodes, in parallel. It compares their results and if there is a disagreement regarding the validity of a certain payload, this is detected and the beacon node is protected from publishing a block with a potential execution layer consensus bug in it. The BN provides metrics per EL node for the number of successful or failed requests for each type Engine API requests. If an EL node goes offline and connectivity is resoted later, we report the problem and the remedy in edge-triggered fashion. * More progress towards implementing Deneb block production in the VC and comparing the value of blocks produced by the EL and the builder API. * Adds a Makefile target for the zhejiang testnet
2023-03-05 01:40:21 +00:00
GETPAYLOAD_TIMEOUT = 1.seconds
Don't report very brief EL connection interruptions on user-visible log levels (#4960)
2023-05-15 20:40:47 +00:00
connectionStateChangeHysteresisThreshold = 15
## How many unsuccesful/successful requests we must see
## before declaring the connection as degraded/restored
Add more flesh to the skeleton :)
2018-11-26 13:33:06 +00:00
type
Spec-compliant implementation of Eth1 monitoring; Eth1-enabled local sim BEWARE! This commit will trigger a stack overflow during local sim
2020-03-24 11:13:07 +00:00
Eth1BlockNumber* = uint64
Eth1BlockTimestamp* = uint64
Support for driving multiple EL nodes from a single Nimbus BN (#4465) * Support for driving multiple EL nodes from a single Nimbus BN Full list of changes: * Eth1Monitor has been renamed to ELManager to match its current responsibilities better. * The ELManager is no longer optional in the code (it won't have a nil value under any circumstances). * The support for subscribing for headers was removed as it only worked with WebSockets and contributed significant complexity while bringing only a very minor advantage. * The `--web3-url` parameter has been deprecated in favor of a new `--el` parameter. The new parameter has a reasonable default value and supports specifying a different JWT for each connection. Each connection can also be configured with a different set of responsibilities (e.g. download deposits, validate blocks and/or produce blocks). On the command-line, these properties can be configured through URL properties stored in the #anchor part of the URL. In TOML files, they come with a very natural syntax (althrough the URL scheme is also supported). * The previously scattered EL-related state and logic is now moved to `eth1_monitor.nim` (this module will be renamed to `el_manager.nim` in a follow-up commit). State is assigned properly either to the `ELManager` or the to individual `ELConnection` objects where appropriate. The ELManager executes all Engine API requests against all attached EL nodes, in parallel. It compares their results and if there is a disagreement regarding the validity of a certain payload, this is detected and the beacon node is protected from publishing a block with a potential execution layer consensus bug in it. The BN provides metrics per EL node for the number of successful or failed requests for each type Engine API requests. If an EL node goes offline and connectivity is resoted later, we report the problem and the remedy in edge-triggered fashion. * More progress towards implementing Deneb block production in the VC and comparing the value of blocks produced by the EL and the builder API. * Adds a Makefile target for the zhejiang testnet
2023-03-05 01:40:21 +00:00
Eth1BlockHeader = engine_api.BlockHeader
Spec-compliant implementation of Eth1 monitoring; Eth1-enabled local sim BEWARE! This commit will trigger a stack overflow during local sim
2020-03-24 11:13:07 +00:00
Eth1Block* = ref object
Minor post-merge cleanups (#3945) https://github.com/status-im/nimbus-eth2/pull/3944 The use of nested `awaitWithRetries` calls would have resulted in an unexpected number of retries (3x3). We now use regular `await` in outer layer to avoid the problem. https://github.com/status-im/nimbus-eth2/pull/3943 The new code has an invariant that the `headMerkleizer` field in the `Eth1Chain` is always kept in sync with the blocks stored in the chain. This invariant is now enforced better by doing the necessary merkleizer updates in the `Eth1Chain.addBlock` function, in the `Eth1Chain.init` function and in the `Eth1Chain.reset` function.
2022-08-10 12:31:10 +00:00
hash*: Eth2Digest
Spec-compliant implementation of Eth1 monitoring; Eth1-enabled local sim BEWARE! This commit will trigger a stack overflow during local sim
2020-03-24 11:13:07 +00:00
number*: Eth1BlockNumber
timestamp*: Eth1BlockTimestamp
Minor post-merge cleanups (#3945) https://github.com/status-im/nimbus-eth2/pull/3944 The use of nested `awaitWithRetries` calls would have resulted in an unexpected number of retries (3x3). We now use regular `await` in outer layer to avoid the problem. https://github.com/status-im/nimbus-eth2/pull/3943 The new code has an invariant that the `headMerkleizer` field in the `Eth1Chain` is always kept in sync with the blocks stored in the chain. This invariant is now enforced better by doing the necessary merkleizer updates in the `Eth1Chain.addBlock` function, in the `Eth1Chain.init` function and in the `Eth1Chain.reset` function.
2022-08-10 12:31:10 +00:00
## Basic properties of the block
## These must be initialized in the constructor
Fix the corrupted database state on Pyrmont nodes; Add mainnet genesis (#2056) * Handle some web3 timeouts better * Add support for developer .env files * Eth1 improvements; Mainnet genesis state Notable changes: * The deposits table have been removed from the database. The client will no longer process all deposits on start-up. * The network metadata now includes a "state snapshot" of the deposit contract. This allows the client to skip syncing deposits made prior to the snapshot (i.e. genesis). Suitable metadata added for Pyrmont and Mainnet. * The Eth1 monitor won't be started unless there are validators attached to the node. * The genesis detection code is now optional and disabled by default * Bugfix: The client should not produce blocks that will fail validation when it hasn't downloaded the latest deposits yet * Bugfix: Work around the database corruption affecting Pyrmont nodes * Remove metadata for Toledo and Medalla
2020-11-24 21:21:47 +00:00
deposits*: seq[DepositData]
Minor post-merge cleanups (#3945) https://github.com/status-im/nimbus-eth2/pull/3944 The use of nested `awaitWithRetries` calls would have resulted in an unexpected number of retries (3x3). We now use regular `await` in outer layer to avoid the problem. https://github.com/status-im/nimbus-eth2/pull/3943 The new code has an invariant that the `headMerkleizer` field in the `Eth1Chain` is always kept in sync with the blocks stored in the chain. This invariant is now enforced better by doing the necessary merkleizer updates in the `Eth1Chain.addBlock` function, in the `Eth1Chain.init` function and in the `Eth1Chain.reset` function.
2022-08-10 12:31:10 +00:00
## Deposits inside this particular block
depositRoot*: Eth2Digest
depositCount*: uint64
## Global deposits count and hash tree root of the entire sequence
## These are computed when the block is added to the chain (see `addBlock`)
Fix the corrupted database state on Pyrmont nodes; Add mainnet genesis (#2056) * Handle some web3 timeouts better * Add support for developer .env files * Eth1 improvements; Mainnet genesis state Notable changes: * The deposits table have been removed from the database. The client will no longer process all deposits on start-up. * The network metadata now includes a "state snapshot" of the deposit contract. This allows the client to skip syncing deposits made prior to the snapshot (i.e. genesis). Suitable metadata added for Pyrmont and Mainnet. * The Eth1 monitor won't be started unless there are validators attached to the node. * The genesis detection code is now optional and disabled by default * Bugfix: The client should not produce blocks that will fail validation when it hasn't downloaded the latest deposits yet * Bugfix: Work around the database corruption affecting Pyrmont nodes * Remove metadata for Toledo and Medalla
2020-11-24 21:21:47 +00:00
Spec-compliant implementation of Eth1 monitoring; Eth1-enabled local sim BEWARE! This commit will trigger a stack overflow during local sim
2020-03-24 11:13:07 +00:00
Eth1Chain* = object
Add Eth1 deposits simulation to block_sim
2020-12-03 04:30:35 +00:00
db: BeaconChainDB
Implement split preset/config support (#2710) * Implement split preset/config support This is the initial bulk refactor to introduce runtime config values in a number of places, somewhat replacing the existing mechanism of loading network metadata. It still needs more work, this is the initial refactor that introduces runtime configuration in some of the places that need it. The PR changes the way presets and constants work, to match the spec. In particular, a "preset" now refers to the compile-time configuration while a "cfg" or "RuntimeConfig" is the dynamic part. A single binary can support either mainnet or minimal, but not both. Support for other presets has been removed completely (can be readded, in case there's need). There's a number of outstanding tasks: * `SECONDS_PER_SLOT` still needs fixing * loading custom runtime configs needs redoing * checking constants against YAML file * yeerongpilly support `build/nimbus_beacon_node --network=yeerongpilly --discv5:no --log-level=DEBUG` * load fork epoch from config * fix fork digest sent in status * nicer error string for request failures * fix tools * one more * fixup * fixup * fixup * use "standard" network definition folder in local testnet Files are loaded from their standard locations, including genesis etc, to conform to the format used in the `eth2-networks` repo. * fix launch scripts, allow unknown config values * fix base config of rest test * cleanups * bundle mainnet config using common loader * fix spec links and names * only include supported preset in binary * drop yeerongpilly, add altair-devnet-0, support boot_enr.yaml
2021-07-12 13:01:38 +00:00
cfg: RuntimeConfig
Add Eth1 deposits simulation to block_sim
2020-12-03 04:30:35 +00:00
finalizedBlockHash: Eth2Digest
finalizedDepositsMerkleizer: DepositsMerkleizer
convert between engine and consensus ExecutionPayloads (#3228) * convert between engine and consensus ExecutionPayloads
2022-01-03 12:22:56 +00:00
## The latest block that reached a 50% majority vote from
## the Eth2 validators according to the follow distance and
## the ETH1_VOTING_PERIOD
avoid materializing potentially long deposits seq (#3947) When fetching eth1 data and deposits for a new block proposal, the list of deposits from previous eth1 data to the next one is fully loaded into a `seq`. This can potentially be a very long list in active periods. Changing this to an `iterator` saves memory by ensuring that the entire list is no longer materialized; only the `DepositData` roots are needed.
2022-08-12 13:52:06 +00:00
blocks*: Deque[Eth1Block]
convert between engine and consensus ExecutionPayloads (#3228) * convert between engine and consensus ExecutionPayloads
2022-01-03 12:22:56 +00:00
## A non-forkable chain of blocks ending at the block with
## ETH1_FOLLOW_DISTANCE offset from the head.
blocksByHash: Table[BlockHash, Eth1Block]
fix obtaining deposits after connection loss (#3943) * fix obtaining deposits after connection loss When an error occurs during Eth1 deposits import, the already imported blocks are kept while the connection to the EL is re-established. However, the corresponding merkleizer is not persisted, leading to any future deposits no longer being properly imported. This is quite common when syncing a fresh Nimbus instance against an already-synced Geth EL. Fixed by persisting the head merkleizer together with the blocks.
2022-08-09 21:32:34 +00:00
headMerkleizer: DepositsMerkleizer
## Merkleizer state after applying all `blocks`
Fix #2261 Also bumps Confutils to allow setting the hidden --web3-mode param (to allow testing the eth1 syncing without validators)
2021-01-29 21:21:44 +00:00
hasConsensusViolation: bool
## The local chain contradicts the observed consensus on the network
Spec-compliant implementation of Eth1 monitoring; Eth1-enabled local sim BEWARE! This commit will trigger a stack overflow during local sim
2020-03-24 11:13:07 +00:00
Support for driving multiple EL nodes from a single Nimbus BN (#4465) * Support for driving multiple EL nodes from a single Nimbus BN Full list of changes: * Eth1Monitor has been renamed to ELManager to match its current responsibilities better. * The ELManager is no longer optional in the code (it won't have a nil value under any circumstances). * The support for subscribing for headers was removed as it only worked with WebSockets and contributed significant complexity while bringing only a very minor advantage. * The `--web3-url` parameter has been deprecated in favor of a new `--el` parameter. The new parameter has a reasonable default value and supports specifying a different JWT for each connection. Each connection can also be configured with a different set of responsibilities (e.g. download deposits, validate blocks and/or produce blocks). On the command-line, these properties can be configured through URL properties stored in the #anchor part of the URL. In TOML files, they come with a very natural syntax (althrough the URL scheme is also supported). * The previously scattered EL-related state and logic is now moved to `eth1_monitor.nim` (this module will be renamed to `el_manager.nim` in a follow-up commit). State is assigned properly either to the `ELManager` or the to individual `ELConnection` objects where appropriate. The ELManager executes all Engine API requests against all attached EL nodes, in parallel. It compares their results and if there is a disagreement regarding the validity of a certain payload, this is detected and the beacon node is protected from publishing a block with a potential execution layer consensus bug in it. The BN provides metrics per EL node for the number of successful or failed requests for each type Engine API requests. If an EL node goes offline and connectivity is resoted later, we report the problem and the remedy in edge-triggered fashion. * More progress towards implementing Deneb block production in the VC and comparing the value of blocks produced by the EL and the builder API. * Adds a Makefile target for the zhejiang testnet
2023-03-05 01:40:21 +00:00
NextExpectedPayloadParams* = object
headBlockHash*: Eth2Digest
safeBlockHash*: Eth2Digest
finalizedBlockHash*: Eth2Digest
use fcU V3 in Deneb/Cancun (#5293)
2023-08-15 23:00:35 +00:00
payloadAttributes*: PayloadAttributesV3
Support for driving multiple EL nodes from a single Nimbus BN (#4465) * Support for driving multiple EL nodes from a single Nimbus BN Full list of changes: * Eth1Monitor has been renamed to ELManager to match its current responsibilities better. * The ELManager is no longer optional in the code (it won't have a nil value under any circumstances). * The support for subscribing for headers was removed as it only worked with WebSockets and contributed significant complexity while bringing only a very minor advantage. * The `--web3-url` parameter has been deprecated in favor of a new `--el` parameter. The new parameter has a reasonable default value and supports specifying a different JWT for each connection. Each connection can also be configured with a different set of responsibilities (e.g. download deposits, validate blocks and/or produce blocks). On the command-line, these properties can be configured through URL properties stored in the #anchor part of the URL. In TOML files, they come with a very natural syntax (althrough the URL scheme is also supported). * The previously scattered EL-related state and logic is now moved to `eth1_monitor.nim` (this module will be renamed to `el_manager.nim` in a follow-up commit). State is assigned properly either to the `ELManager` or the to individual `ELConnection` objects where appropriate. The ELManager executes all Engine API requests against all attached EL nodes, in parallel. It compares their results and if there is a disagreement regarding the validity of a certain payload, this is detected and the beacon node is protected from publishing a block with a potential execution layer consensus bug in it. The BN provides metrics per EL node for the number of successful or failed requests for each type Engine API requests. If an EL node goes offline and connectivity is resoted later, we report the problem and the remedy in edge-triggered fashion. * More progress towards implementing Deneb block production in the VC and comparing the value of blocks produced by the EL and the builder API. * Adds a Makefile target for the zhejiang testnet
2023-03-05 01:40:21 +00:00
ELManager* = ref object
More complete reset of the web3 provider on each failure; Fix #2184
2020-12-15 21:59:29 +00:00
eth1Network: Option[Eth1Network]
Rename eth1/ -> el/ and eth1_monitor.nim -> el_monitor.nim (#4944)
2023-05-15 05:05:12 +00:00
## If this value is supplied the EL manager will check whether
Support for driving multiple EL nodes from a single Nimbus BN (#4465) * Support for driving multiple EL nodes from a single Nimbus BN Full list of changes: * Eth1Monitor has been renamed to ELManager to match its current responsibilities better. * The ELManager is no longer optional in the code (it won't have a nil value under any circumstances). * The support for subscribing for headers was removed as it only worked with WebSockets and contributed significant complexity while bringing only a very minor advantage. * The `--web3-url` parameter has been deprecated in favor of a new `--el` parameter. The new parameter has a reasonable default value and supports specifying a different JWT for each connection. Each connection can also be configured with a different set of responsibilities (e.g. download deposits, validate blocks and/or produce blocks). On the command-line, these properties can be configured through URL properties stored in the #anchor part of the URL. In TOML files, they come with a very natural syntax (althrough the URL scheme is also supported). * The previously scattered EL-related state and logic is now moved to `eth1_monitor.nim` (this module will be renamed to `el_manager.nim` in a follow-up commit). State is assigned properly either to the `ELManager` or the to individual `ELConnection` objects where appropriate. The ELManager executes all Engine API requests against all attached EL nodes, in parallel. It compares their results and if there is a disagreement regarding the validity of a certain payload, this is detected and the beacon node is protected from publishing a block with a potential execution layer consensus bug in it. The BN provides metrics per EL node for the number of successful or failed requests for each type Engine API requests. If an EL node goes offline and connectivity is resoted later, we report the problem and the remedy in edge-triggered fashion. * More progress towards implementing Deneb block production in the VC and comparing the value of blocks produced by the EL and the builder API. * Adds a Makefile target for the zhejiang testnet
2023-03-05 01:40:21 +00:00
## all configured EL nodes are connected to the same network.
More complete reset of the web3 provider on each failure; Fix #2184
2020-12-15 21:59:29 +00:00
depositContractAddress*: Eth1Address
Support for driving multiple EL nodes from a single Nimbus BN (#4465) * Support for driving multiple EL nodes from a single Nimbus BN Full list of changes: * Eth1Monitor has been renamed to ELManager to match its current responsibilities better. * The ELManager is no longer optional in the code (it won't have a nil value under any circumstances). * The support for subscribing for headers was removed as it only worked with WebSockets and contributed significant complexity while bringing only a very minor advantage. * The `--web3-url` parameter has been deprecated in favor of a new `--el` parameter. The new parameter has a reasonable default value and supports specifying a different JWT for each connection. Each connection can also be configured with a different set of responsibilities (e.g. download deposits, validate blocks and/or produce blocks). On the command-line, these properties can be configured through URL properties stored in the #anchor part of the URL. In TOML files, they come with a very natural syntax (althrough the URL scheme is also supported). * The previously scattered EL-related state and logic is now moved to `eth1_monitor.nim` (this module will be renamed to `el_manager.nim` in a follow-up commit). State is assigned properly either to the `ELManager` or the to individual `ELConnection` objects where appropriate. The ELManager executes all Engine API requests against all attached EL nodes, in parallel. It compares their results and if there is a disagreement regarding the validity of a certain payload, this is detected and the beacon node is protected from publishing a block with a potential execution layer consensus bug in it. The BN provides metrics per EL node for the number of successful or failed requests for each type Engine API requests. If an EL node goes offline and connectivity is resoted later, we report the problem and the remedy in edge-triggered fashion. * More progress towards implementing Deneb block production in the VC and comparing the value of blocks produced by the EL and the builder API. * Adds a Makefile target for the zhejiang testnet
2023-03-05 01:40:21 +00:00
depositContractBlockNumber: uint64
depositContractBlockHash: BlockHash
track request chunk size across EL reconnects (#3960) When the EL connection is interrupted, deposits are once more requested in chunks of 5000 blocks. This is a problem when the response takes over a minute to produce and consistently times out as followup requests with lower chunk sizes may no longer work after a request was canceled, e.g., when using Geth with websockets. By keeping track of `blocksPerRequest` across EL reconnections, it is possible to recover from this by avoiding to continuously repeat the initial request with the full 5000 blocks. Also cleans up one more "retry of retry" instance; `DataProviderTimeout` is a `CatchableError` and already handled by the existing retry logic.
2022-08-12 13:51:33 +00:00
blocksPerLogsRequest: uint64
Support for driving multiple EL nodes from a single Nimbus BN (#4465) * Support for driving multiple EL nodes from a single Nimbus BN Full list of changes: * Eth1Monitor has been renamed to ELManager to match its current responsibilities better. * The ELManager is no longer optional in the code (it won't have a nil value under any circumstances). * The support for subscribing for headers was removed as it only worked with WebSockets and contributed significant complexity while bringing only a very minor advantage. * The `--web3-url` parameter has been deprecated in favor of a new `--el` parameter. The new parameter has a reasonable default value and supports specifying a different JWT for each connection. Each connection can also be configured with a different set of responsibilities (e.g. download deposits, validate blocks and/or produce blocks). On the command-line, these properties can be configured through URL properties stored in the #anchor part of the URL. In TOML files, they come with a very natural syntax (althrough the URL scheme is also supported). * The previously scattered EL-related state and logic is now moved to `eth1_monitor.nim` (this module will be renamed to `el_manager.nim` in a follow-up commit). State is assigned properly either to the `ELManager` or the to individual `ELConnection` objects where appropriate. The ELManager executes all Engine API requests against all attached EL nodes, in parallel. It compares their results and if there is a disagreement regarding the validity of a certain payload, this is detected and the beacon node is protected from publishing a block with a potential execution layer consensus bug in it. The BN provides metrics per EL node for the number of successful or failed requests for each type Engine API requests. If an EL node goes offline and connectivity is resoted later, we report the problem and the remedy in edge-triggered fashion. * More progress towards implementing Deneb block production in the VC and comparing the value of blocks produced by the EL and the builder API. * Adds a Makefile target for the zhejiang testnet
2023-03-05 01:40:21 +00:00
## This value is used to dynamically adjust the number of
## blocks we are trying to download at once during deposit
## syncing. By default, the value is set to the constant
## `targetBlocksPerLogsRequest`, but if the EL is failing
## to serve this number of blocks per single `eth_getLogs`
## request, we temporarily lower the value until the request
## succeeds. The failures are generally expected only in
## periods in the history for very high deposit density.
elConnections: seq[ELConnection]
## All active EL connections
eth1Chain: Eth1Chain
## At larger distances, this chain consists of all blocks
## with deposits. Within the relevant voting period, it
## also includes blocks without deposits because we must
## vote for a block only if it's part of our known history.
syncTargetBlock: Option[Eth1BlockNumber]
chainSyncingLoopFut: Future[void]
exchangeTransitionConfigurationLoopFut: Future[void]
stopFut: Future[void]
Simplify the mainchain monitor
2020-10-14 14:04:08 +00:00
Support for driving multiple EL nodes from a single Nimbus BN (#4465) * Support for driving multiple EL nodes from a single Nimbus BN Full list of changes: * Eth1Monitor has been renamed to ELManager to match its current responsibilities better. * The ELManager is no longer optional in the code (it won't have a nil value under any circumstances). * The support for subscribing for headers was removed as it only worked with WebSockets and contributed significant complexity while bringing only a very minor advantage. * The `--web3-url` parameter has been deprecated in favor of a new `--el` parameter. The new parameter has a reasonable default value and supports specifying a different JWT for each connection. Each connection can also be configured with a different set of responsibilities (e.g. download deposits, validate blocks and/or produce blocks). On the command-line, these properties can be configured through URL properties stored in the #anchor part of the URL. In TOML files, they come with a very natural syntax (althrough the URL scheme is also supported). * The previously scattered EL-related state and logic is now moved to `eth1_monitor.nim` (this module will be renamed to `el_manager.nim` in a follow-up commit). State is assigned properly either to the `ELManager` or the to individual `ELConnection` objects where appropriate. The ELManager executes all Engine API requests against all attached EL nodes, in parallel. It compares their results and if there is a disagreement regarding the validity of a certain payload, this is detected and the beacon node is protected from publishing a block with a potential execution layer consensus bug in it. The BN provides metrics per EL node for the number of successful or failed requests for each type Engine API requests. If an EL node goes offline and connectivity is resoted later, we report the problem and the remedy in edge-triggered fashion. * More progress towards implementing Deneb block production in the VC and comparing the value of blocks produced by the EL and the builder API. * Adds a Makefile target for the zhejiang testnet
2023-03-05 01:40:21 +00:00
nextExpectedPayloadParams*: Option[NextExpectedPayloadParams]
Further reduce the number of RPC requests; Significantly faster ETH1 syncing
2020-11-03 01:21:07 +00:00
Support for driving multiple EL nodes from a single Nimbus BN (#4465) * Support for driving multiple EL nodes from a single Nimbus BN Full list of changes: * Eth1Monitor has been renamed to ELManager to match its current responsibilities better. * The ELManager is no longer optional in the code (it won't have a nil value under any circumstances). * The support for subscribing for headers was removed as it only worked with WebSockets and contributed significant complexity while bringing only a very minor advantage. * The `--web3-url` parameter has been deprecated in favor of a new `--el` parameter. The new parameter has a reasonable default value and supports specifying a different JWT for each connection. Each connection can also be configured with a different set of responsibilities (e.g. download deposits, validate blocks and/or produce blocks). On the command-line, these properties can be configured through URL properties stored in the #anchor part of the URL. In TOML files, they come with a very natural syntax (althrough the URL scheme is also supported). * The previously scattered EL-related state and logic is now moved to `eth1_monitor.nim` (this module will be renamed to `el_manager.nim` in a follow-up commit). State is assigned properly either to the `ELManager` or the to individual `ELConnection` objects where appropriate. The ELManager executes all Engine API requests against all attached EL nodes, in parallel. It compares their results and if there is a disagreement regarding the validity of a certain payload, this is detected and the beacon node is protected from publishing a block with a potential execution layer consensus bug in it. The BN provides metrics per EL node for the number of successful or failed requests for each type Engine API requests. If an EL node goes offline and connectivity is resoted later, we report the problem and the remedy in edge-triggered fashion. * More progress towards implementing Deneb block production in the VC and comparing the value of blocks produced by the EL and the builder API. * Adds a Makefile target for the zhejiang testnet
2023-03-05 01:40:21 +00:00
EtcStatus {.pure.} = enum
notExchangedYet
mismatch
match
Further reduce the number of RPC requests; Significantly faster ETH1 syncing
2020-11-03 01:21:07 +00:00
Support for driving multiple EL nodes from a single Nimbus BN (#4465) * Support for driving multiple EL nodes from a single Nimbus BN Full list of changes: * Eth1Monitor has been renamed to ELManager to match its current responsibilities better. * The ELManager is no longer optional in the code (it won't have a nil value under any circumstances). * The support for subscribing for headers was removed as it only worked with WebSockets and contributed significant complexity while bringing only a very minor advantage. * The `--web3-url` parameter has been deprecated in favor of a new `--el` parameter. The new parameter has a reasonable default value and supports specifying a different JWT for each connection. Each connection can also be configured with a different set of responsibilities (e.g. download deposits, validate blocks and/or produce blocks). On the command-line, these properties can be configured through URL properties stored in the #anchor part of the URL. In TOML files, they come with a very natural syntax (althrough the URL scheme is also supported). * The previously scattered EL-related state and logic is now moved to `eth1_monitor.nim` (this module will be renamed to `el_manager.nim` in a follow-up commit). State is assigned properly either to the `ELManager` or the to individual `ELConnection` objects where appropriate. The ELManager executes all Engine API requests against all attached EL nodes, in parallel. It compares their results and if there is a disagreement regarding the validity of a certain payload, this is detected and the beacon node is protected from publishing a block with a potential execution layer consensus bug in it. The BN provides metrics per EL node for the number of successful or failed requests for each type Engine API requests. If an EL node goes offline and connectivity is resoted later, we report the problem and the remedy in edge-triggered fashion. * More progress towards implementing Deneb block production in the VC and comparing the value of blocks produced by the EL and the builder API. * Adds a Makefile target for the zhejiang testnet
2023-03-05 01:40:21 +00:00
DepositContractSyncStatus {.pure.} = enum
unknown
notSynced
synced
Bellatrix TTD detection (#3745) * Bellatrix TTD detection * Update beacon_chain/eth1/eth1_monitor.nim Co-authored-by: Etan Kissling <etan@status.im> * Update beacon_chain/nimbus_beacon_node.nim Co-authored-by: tersec <tersec@users.noreply.github.com> Co-authored-by: Etan Kissling <etan@status.im> Co-authored-by: tersec <tersec@users.noreply.github.com>
2022-06-15 02:38:27 +00:00
Support for driving multiple EL nodes from a single Nimbus BN (#4465) * Support for driving multiple EL nodes from a single Nimbus BN Full list of changes: * Eth1Monitor has been renamed to ELManager to match its current responsibilities better. * The ELManager is no longer optional in the code (it won't have a nil value under any circumstances). * The support for subscribing for headers was removed as it only worked with WebSockets and contributed significant complexity while bringing only a very minor advantage. * The `--web3-url` parameter has been deprecated in favor of a new `--el` parameter. The new parameter has a reasonable default value and supports specifying a different JWT for each connection. Each connection can also be configured with a different set of responsibilities (e.g. download deposits, validate blocks and/or produce blocks). On the command-line, these properties can be configured through URL properties stored in the #anchor part of the URL. In TOML files, they come with a very natural syntax (althrough the URL scheme is also supported). * The previously scattered EL-related state and logic is now moved to `eth1_monitor.nim` (this module will be renamed to `el_manager.nim` in a follow-up commit). State is assigned properly either to the `ELManager` or the to individual `ELConnection` objects where appropriate. The ELManager executes all Engine API requests against all attached EL nodes, in parallel. It compares their results and if there is a disagreement regarding the validity of a certain payload, this is detected and the beacon node is protected from publishing a block with a potential execution layer consensus bug in it. The BN provides metrics per EL node for the number of successful or failed requests for each type Engine API requests. If an EL node goes offline and connectivity is resoted later, we report the problem and the remedy in edge-triggered fashion. * More progress towards implementing Deneb block production in the VC and comparing the value of blocks produced by the EL and the builder API. * Adds a Makefile target for the zhejiang testnet
2023-03-05 01:40:21 +00:00
ConnectionState = enum
NeverTested
Working
Degraded
ELConnection* = ref object
engineUrl: EngineApiUrl
web3: Option[Web3]
## This will be `none` before connecting and while we are
## reconnecting after a lost connetion. You can wait on
## the future below for the moment the connection is active.
connectingFut: Future[Result[Web3, string]]
## This future will be replaced when the connection is lost.
etcStatus: EtcStatus
## The latest status of the `exchangeTransitionConfiguration`
## exchange.
state: ConnectionState
Don't report very brief EL connection interruptions on user-visible log levels (#4960)
2023-05-15 20:40:47 +00:00
hysteresisCounter: int
Better connection error handling in MainchainMonitor
2019-11-22 13:16:07 +00:00
Support for driving multiple EL nodes from a single Nimbus BN (#4465) * Support for driving multiple EL nodes from a single Nimbus BN Full list of changes: * Eth1Monitor has been renamed to ELManager to match its current responsibilities better. * The ELManager is no longer optional in the code (it won't have a nil value under any circumstances). * The support for subscribing for headers was removed as it only worked with WebSockets and contributed significant complexity while bringing only a very minor advantage. * The `--web3-url` parameter has been deprecated in favor of a new `--el` parameter. The new parameter has a reasonable default value and supports specifying a different JWT for each connection. Each connection can also be configured with a different set of responsibilities (e.g. download deposits, validate blocks and/or produce blocks). On the command-line, these properties can be configured through URL properties stored in the #anchor part of the URL. In TOML files, they come with a very natural syntax (althrough the URL scheme is also supported). * The previously scattered EL-related state and logic is now moved to `eth1_monitor.nim` (this module will be renamed to `el_manager.nim` in a follow-up commit). State is assigned properly either to the `ELManager` or the to individual `ELConnection` objects where appropriate. The ELManager executes all Engine API requests against all attached EL nodes, in parallel. It compares their results and if there is a disagreement regarding the validity of a certain payload, this is detected and the beacon node is protected from publishing a block with a potential execution layer consensus bug in it. The BN provides metrics per EL node for the number of successful or failed requests for each type Engine API requests. If an EL node goes offline and connectivity is resoted later, we report the problem and the remedy in edge-triggered fashion. * More progress towards implementing Deneb block production in the VC and comparing the value of blocks produced by the EL and the builder API. * Adds a Makefile target for the zhejiang testnet
2023-03-05 01:40:21 +00:00
depositContractSyncStatus: DepositContractSyncStatus
## Are we sure that this EL has synced the deposit contract?
Spec-compliant implementation of Eth1 monitoring; Eth1-enabled local sim BEWARE! This commit will trigger a stack overflow during local sim
2020-03-24 11:13:07 +00:00
Support for driving multiple EL nodes from a single Nimbus BN (#4465) * Support for driving multiple EL nodes from a single Nimbus BN Full list of changes: * Eth1Monitor has been renamed to ELManager to match its current responsibilities better. * The ELManager is no longer optional in the code (it won't have a nil value under any circumstances). * The support for subscribing for headers was removed as it only worked with WebSockets and contributed significant complexity while bringing only a very minor advantage. * The `--web3-url` parameter has been deprecated in favor of a new `--el` parameter. The new parameter has a reasonable default value and supports specifying a different JWT for each connection. Each connection can also be configured with a different set of responsibilities (e.g. download deposits, validate blocks and/or produce blocks). On the command-line, these properties can be configured through URL properties stored in the #anchor part of the URL. In TOML files, they come with a very natural syntax (althrough the URL scheme is also supported). * The previously scattered EL-related state and logic is now moved to `eth1_monitor.nim` (this module will be renamed to `el_manager.nim` in a follow-up commit). State is assigned properly either to the `ELManager` or the to individual `ELConnection` objects where appropriate. The ELManager executes all Engine API requests against all attached EL nodes, in parallel. It compares their results and if there is a disagreement regarding the validity of a certain payload, this is detected and the beacon node is protected from publishing a block with a potential execution layer consensus bug in it. The BN provides metrics per EL node for the number of successful or failed requests for each type Engine API requests. If an EL node goes offline and connectivity is resoted later, we report the problem and the remedy in edge-triggered fashion. * More progress towards implementing Deneb block production in the VC and comparing the value of blocks produced by the EL and the builder API. * Adds a Makefile target for the zhejiang testnet
2023-03-05 01:40:21 +00:00
lastPayloadId: Option[engine_api.PayloadID]
Spec-compliant implementation of Eth1 monitoring; Eth1-enabled local sim BEWARE! This commit will trigger a stack overflow during local sim
2020-03-24 11:13:07 +00:00
Add polling support in the Eth1Monitor (extracted from the merge branch)
2021-11-25 16:51:51 +00:00
FullBlockId* = object
number: Eth1BlockNumber
hash: BlockHash
A simple script for fetching detailed deposit data
2021-05-28 12:42:39 +00:00
DataProviderFailure* = object of CatchableError
CorruptDataProvider* = object of DataProviderFailure
DataProviderTimeout* = object of DataProviderFailure
Spec-compliant implementation of Eth1 monitoring; Eth1-enabled local sim BEWARE! This commit will trigger a stack overflow during local sim
2020-03-24 11:13:07 +00:00
remove `{.raises: [Defect].}` Nim 1.2 compatibility (#5352) In Nim 1.6, `{.raises: [Defect].}` is no longer necessary. Remove it.
2023-08-25 09:29:07 +00:00
DisconnectHandler* = proc () {.gcsafe, raises: [].}
Spec-compliant implementation of Eth1 monitoring; Eth1-enabled local sim BEWARE! This commit will trigger a stack overflow during local sim
2020-03-24 11:13:07 +00:00
DepositEventHandler* = proc (
The deposit contract uses dynamic bytes for most of its event properties https://github.com/ethereum/consensus-specs/blob/dev/solidity_deposit_contract/deposit_contract.sol#L19-L24
2021-10-14 12:33:55 +00:00
pubkey: PubKeyBytes,
withdrawalCredentials: WithdrawalCredentialsBytes,
amount: Int64LeBytes,
signature: SignatureBytes,
merkleTreeIndex: Int64LeBytes,
remove `{.raises: [Defect].}` Nim 1.2 compatibility (#5352) In Nim 1.6, `{.raises: [Defect].}` is no longer necessary. Remove it.
2023-08-25 09:29:07 +00:00
j: JsonNode) {.gcsafe, raises: [].}
Spec-compliant implementation of Eth1 monitoring; Eth1-enabled local sim BEWARE! This commit will trigger a stack overflow during local sim
2020-03-24 11:13:07 +00:00
Tentative fix for the Pyrmont Eth1 voting issue
2020-11-19 17:19:03 +00:00
BlockProposalEth1Data* = object
vote*: Eth1Data
deposits*: seq[Deposit]
Fix the corrupted database state on Pyrmont nodes; Add mainnet genesis (#2056) * Handle some web3 timeouts better * Add support for developer .env files * Eth1 improvements; Mainnet genesis state Notable changes: * The deposits table have been removed from the database. The client will no longer process all deposits on start-up. * The network metadata now includes a "state snapshot" of the deposit contract. This allows the client to skip syncing deposits made prior to the snapshot (i.e. genesis). Suitable metadata added for Pyrmont and Mainnet. * The Eth1 monitor won't be started unless there are validators attached to the node. * The genesis detection code is now optional and disabled by default * Bugfix: The client should not produce blocks that will fail validation when it hasn't downloaded the latest deposits yet * Bugfix: Work around the database corruption affecting Pyrmont nodes * Remove metadata for Toledo and Medalla
2020-11-24 21:21:47 +00:00
hasMissingDeposits*: bool
Tentative fix for the Pyrmont Eth1 voting issue
2020-11-19 17:19:03 +00:00
Support for driving multiple EL nodes from a single Nimbus BN (#4465) * Support for driving multiple EL nodes from a single Nimbus BN Full list of changes: * Eth1Monitor has been renamed to ELManager to match its current responsibilities better. * The ELManager is no longer optional in the code (it won't have a nil value under any circumstances). * The support for subscribing for headers was removed as it only worked with WebSockets and contributed significant complexity while bringing only a very minor advantage. * The `--web3-url` parameter has been deprecated in favor of a new `--el` parameter. The new parameter has a reasonable default value and supports specifying a different JWT for each connection. Each connection can also be configured with a different set of responsibilities (e.g. download deposits, validate blocks and/or produce blocks). On the command-line, these properties can be configured through URL properties stored in the #anchor part of the URL. In TOML files, they come with a very natural syntax (althrough the URL scheme is also supported). * The previously scattered EL-related state and logic is now moved to `eth1_monitor.nim` (this module will be renamed to `el_manager.nim` in a follow-up commit). State is assigned properly either to the `ELManager` or the to individual `ELConnection` objects where appropriate. The ELManager executes all Engine API requests against all attached EL nodes, in parallel. It compares their results and if there is a disagreement regarding the validity of a certain payload, this is detected and the beacon node is protected from publishing a block with a potential execution layer consensus bug in it. The BN provides metrics per EL node for the number of successful or failed requests for each type Engine API requests. If an EL node goes offline and connectivity is resoted later, we report the problem and the remedy in edge-triggered fashion. * More progress towards implementing Deneb block production in the VC and comparing the value of blocks produced by the EL and the builder API. * Adds a Makefile target for the zhejiang testnet
2023-03-05 01:40:21 +00:00
BellatrixExecutionPayloadWithValue* = object
executionPayload*: ExecutionPayloadV1
blockValue*: UInt256
SomeEnginePayloadWithValue =
BellatrixExecutionPayloadWithValue |
GetPayloadV2Response |
Update nim-web3 (#4875) Update to new version of nim-web3 incorporating the execution API change that merges getPayloadV3 and getBlobsBundleV1.
2023-04-28 19:20:25 +00:00
GetPayloadV3Response
Support for driving multiple EL nodes from a single Nimbus BN (#4465) * Support for driving multiple EL nodes from a single Nimbus BN Full list of changes: * Eth1Monitor has been renamed to ELManager to match its current responsibilities better. * The ELManager is no longer optional in the code (it won't have a nil value under any circumstances). * The support for subscribing for headers was removed as it only worked with WebSockets and contributed significant complexity while bringing only a very minor advantage. * The `--web3-url` parameter has been deprecated in favor of a new `--el` parameter. The new parameter has a reasonable default value and supports specifying a different JWT for each connection. Each connection can also be configured with a different set of responsibilities (e.g. download deposits, validate blocks and/or produce blocks). On the command-line, these properties can be configured through URL properties stored in the #anchor part of the URL. In TOML files, they come with a very natural syntax (althrough the URL scheme is also supported). * The previously scattered EL-related state and logic is now moved to `eth1_monitor.nim` (this module will be renamed to `el_manager.nim` in a follow-up commit). State is assigned properly either to the `ELManager` or the to individual `ELConnection` objects where appropriate. The ELManager executes all Engine API requests against all attached EL nodes, in parallel. It compares their results and if there is a disagreement regarding the validity of a certain payload, this is detected and the beacon node is protected from publishing a block with a potential execution layer consensus bug in it. The BN provides metrics per EL node for the number of successful or failed requests for each type Engine API requests. If an EL node goes offline and connectivity is resoted later, we report the problem and the remedy in edge-triggered fashion. * More progress towards implementing Deneb block production in the VC and comparing the value of blocks produced by the EL and the builder API. * Adds a Makefile target for the zhejiang testnet
2023-03-05 01:40:21 +00:00
Add some useful Eth1 monitor metrics
2020-12-09 22:44:59 +00:00
declareCounter failed_web3_requests,
"Failed web3 requests"
declareGauge eth1_latest_head,
"The highest Eth1 block number observed on the network"
declareGauge eth1_synced_head,
"Block number of the highest synchronized block according to follow distance"
declareGauge eth1_finalized_head,
"Block number of the highest Eth1 block finalized by Eth2 consensus"
declareGauge eth1_finalized_deposits,
"Number of deposits that were finalized by the Eth2 consensus"
declareGauge eth1_chain_len,
"The length of the in-memory chain of Eth1 blocks"
Support for driving multiple EL nodes from a single Nimbus BN (#4465) * Support for driving multiple EL nodes from a single Nimbus BN Full list of changes: * Eth1Monitor has been renamed to ELManager to match its current responsibilities better. * The ELManager is no longer optional in the code (it won't have a nil value under any circumstances). * The support for subscribing for headers was removed as it only worked with WebSockets and contributed significant complexity while bringing only a very minor advantage. * The `--web3-url` parameter has been deprecated in favor of a new `--el` parameter. The new parameter has a reasonable default value and supports specifying a different JWT for each connection. Each connection can also be configured with a different set of responsibilities (e.g. download deposits, validate blocks and/or produce blocks). On the command-line, these properties can be configured through URL properties stored in the #anchor part of the URL. In TOML files, they come with a very natural syntax (althrough the URL scheme is also supported). * The previously scattered EL-related state and logic is now moved to `eth1_monitor.nim` (this module will be renamed to `el_manager.nim` in a follow-up commit). State is assigned properly either to the `ELManager` or the to individual `ELConnection` objects where appropriate. The ELManager executes all Engine API requests against all attached EL nodes, in parallel. It compares their results and if there is a disagreement regarding the validity of a certain payload, this is detected and the beacon node is protected from publishing a block with a potential execution layer consensus bug in it. The BN provides metrics per EL node for the number of successful or failed requests for each type Engine API requests. If an EL node goes offline and connectivity is resoted later, we report the problem and the remedy in edge-triggered fashion. * More progress towards implementing Deneb block production in the VC and comparing the value of blocks produced by the EL and the builder API. * Adds a Makefile target for the zhejiang testnet
2023-03-05 01:40:21 +00:00
declareCounter engine_api_responses,
"Number of successful requests to the newPayload Engine API end-point",
labels = ["url", "request", "status"]
engine_api_response_time renamed to engine_api_request_duration_seconds
2023-03-09 17:24:03 +00:00
declareHistogram engine_api_request_duration_seconds,
More metrics for the EL monitor (#4707) * `engine_api_response_time` provides a histogram for the Engine API response times for each unique pair ot URL and request type. * All engine API requests are now tracked Other changes: The client will no longer exit on start-up if it fails to connect to a properly configured EL node.
2023-03-09 17:29:04 +00:00
"Time(s) used to generate signature usign remote signer",
engine_api_response_time renamed to engine_api_request_duration_seconds
2023-03-09 17:24:03 +00:00
buckets = [0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1.0, 2.5, 5.0],
More metrics for the EL monitor (#4707) * `engine_api_response_time` provides a histogram for the Engine API response times for each unique pair ot URL and request type. * All engine API requests are now tracked Other changes: The client will no longer exit on start-up if it fails to connect to a properly configured EL node.
2023-03-09 17:29:04 +00:00
labels = ["url", "request"]
Support for driving multiple EL nodes from a single Nimbus BN (#4465) * Support for driving multiple EL nodes from a single Nimbus BN Full list of changes: * Eth1Monitor has been renamed to ELManager to match its current responsibilities better. * The ELManager is no longer optional in the code (it won't have a nil value under any circumstances). * The support for subscribing for headers was removed as it only worked with WebSockets and contributed significant complexity while bringing only a very minor advantage. * The `--web3-url` parameter has been deprecated in favor of a new `--el` parameter. The new parameter has a reasonable default value and supports specifying a different JWT for each connection. Each connection can also be configured with a different set of responsibilities (e.g. download deposits, validate blocks and/or produce blocks). On the command-line, these properties can be configured through URL properties stored in the #anchor part of the URL. In TOML files, they come with a very natural syntax (althrough the URL scheme is also supported). * The previously scattered EL-related state and logic is now moved to `eth1_monitor.nim` (this module will be renamed to `el_manager.nim` in a follow-up commit). State is assigned properly either to the `ELManager` or the to individual `ELConnection` objects where appropriate. The ELManager executes all Engine API requests against all attached EL nodes, in parallel. It compares their results and if there is a disagreement regarding the validity of a certain payload, this is detected and the beacon node is protected from publishing a block with a potential execution layer consensus bug in it. The BN provides metrics per EL node for the number of successful or failed requests for each type Engine API requests. If an EL node goes offline and connectivity is resoted later, we report the problem and the remedy in edge-triggered fashion. * More progress towards implementing Deneb block production in the VC and comparing the value of blocks produced by the EL and the builder API. * Adds a Makefile target for the zhejiang testnet
2023-03-05 01:40:21 +00:00
declareCounter engine_api_timeouts,
"Number of timed-out requests to Engine API end-point",
labels = ["url", "request"]
declareCounter engine_api_last_minute_forkchoice_updates_sent,
"Number of last minute requests to the forkchoiceUpdated Engine API end-point just before block proposals",
labels = ["url"]
Don't report very brief EL connection interruptions on user-visible log levels (#4960)
2023-05-15 20:40:47 +00:00
proc close(connection: ELConnection): Future[void] {.async.} =
if connection.web3.isSome:
awaitWithTimeout(connection.web3.get.close(), 30.seconds):
debug "Failed to close data provider in time"
proc increaseCounterTowardsStateChange(connection: ELConnection): bool =
result = connection.hysteresisCounter >= connectionStateChangeHysteresisThreshold
if result:
connection.hysteresisCounter = 0
else:
inc connection.hysteresisCounter
proc decreaseCounterTowardsStateChange(connection: ELConnection) =
if connection.hysteresisCounter > 0:
# While we increase the counter by 1, we decreate it by 20% in order
# to require a steady and affirmative change instead of allowing
# the counter to drift very slowly in one direction when the ratio
# between success and failure is roughly 50:50%
connection.hysteresisCounter = connection.hysteresisCounter div 5
More metrics for the EL monitor (#4707) * `engine_api_response_time` provides a histogram for the Engine API response times for each unique pair ot URL and request type. * All engine API requests are now tracked Other changes: The client will no longer exit on start-up if it fails to connect to a properly configured EL node.
2023-03-09 17:29:04 +00:00
proc setDegradedState(connection: ELConnection,
requestName: string,
statusCode: int, errMsg: string) =
Don't report very brief EL connection interruptions on user-visible log levels (#4960)
2023-05-15 20:40:47 +00:00
debug "Failed EL Request", requestName, statusCode, err = errMsg
More metrics for the EL monitor (#4707) * `engine_api_response_time` provides a histogram for the Engine API response times for each unique pair ot URL and request type. * All engine API requests are now tracked Other changes: The client will no longer exit on start-up if it fails to connect to a properly configured EL node.
2023-03-09 17:29:04 +00:00
case connection.state
of NeverTested, Working:
Don't report very brief EL connection interruptions on user-visible log levels (#4960)
2023-05-15 20:40:47 +00:00
if connection.increaseCounterTowardsStateChange():
warn "Connection to EL node degraded",
url = url(connection.engineUrl),
failedRequest = requestName,
statusCode, err = errMsg
connection.state = Degraded
More forceful EL connection resetting (fixes #4424) (#4708)
2023-03-09 23:41:28 +00:00
Don't report very brief EL connection interruptions on user-visible log levels (#4960)
2023-05-15 20:40:47 +00:00
asyncSpawn connection.close()
connection.web3 = none[Web3]()
of Degraded:
connection.decreaseCounterTowardsStateChange()
More metrics for the EL monitor (#4707) * `engine_api_response_time` provides a histogram for the Engine API response times for each unique pair ot URL and request type. * All engine API requests are now tracked Other changes: The client will no longer exit on start-up if it fails to connect to a properly configured EL node.
2023-03-09 17:29:04 +00:00
proc setWorkingState(connection: ELConnection) =
case connection.state
allow `NeverTested` > `Working` transition for EL connection (#4991) Since #4960, the EL connection status can no longer transition from `NeverTested` to `Working`. Fix that, and also consider `NeverTested` connections as online for the purpose of the `el_offline` REST response.
2023-05-25 07:39:47 +00:00
of NeverTested:
connection.hysteresisCounter = 0
connection.state = Working
More metrics for the EL monitor (#4707) * `engine_api_response_time` provides a histogram for the Engine API response times for each unique pair ot URL and request type. * All engine API requests are now tracked Other changes: The client will no longer exit on start-up if it fails to connect to a properly configured EL node.
2023-03-09 17:29:04 +00:00
of Degraded:
Don't report very brief EL connection interruptions on user-visible log levels (#4960)
2023-05-15 20:40:47 +00:00
if connection.increaseCounterTowardsStateChange():
info "Connection to EL node restored",
url = url(connection.engineUrl)
connection.state = Working
allow `NeverTested` > `Working` transition for EL connection (#4991) Since #4960, the EL connection status can no longer transition from `NeverTested` to `Working`. Fix that, and also consider `NeverTested` connections as online for the purpose of the `el_offline` REST response.
2023-05-25 07:39:47 +00:00
of Working:
Don't report very brief EL connection interruptions on user-visible log levels (#4960)
2023-05-15 20:40:47 +00:00
connection.decreaseCounterTowardsStateChange()
More metrics for the EL monitor (#4707) * `engine_api_response_time` provides a histogram for the Engine API response times for each unique pair ot URL and request type. * All engine API requests are now tracked Other changes: The client will no longer exit on start-up if it fails to connect to a properly configured EL node.
2023-03-09 17:29:04 +00:00
Support for driving multiple EL nodes from a single Nimbus BN (#4465) * Support for driving multiple EL nodes from a single Nimbus BN Full list of changes: * Eth1Monitor has been renamed to ELManager to match its current responsibilities better. * The ELManager is no longer optional in the code (it won't have a nil value under any circumstances). * The support for subscribing for headers was removed as it only worked with WebSockets and contributed significant complexity while bringing only a very minor advantage. * The `--web3-url` parameter has been deprecated in favor of a new `--el` parameter. The new parameter has a reasonable default value and supports specifying a different JWT for each connection. Each connection can also be configured with a different set of responsibilities (e.g. download deposits, validate blocks and/or produce blocks). On the command-line, these properties can be configured through URL properties stored in the #anchor part of the URL. In TOML files, they come with a very natural syntax (althrough the URL scheme is also supported). * The previously scattered EL-related state and logic is now moved to `eth1_monitor.nim` (this module will be renamed to `el_manager.nim` in a follow-up commit). State is assigned properly either to the `ELManager` or the to individual `ELConnection` objects where appropriate. The ELManager executes all Engine API requests against all attached EL nodes, in parallel. It compares their results and if there is a disagreement regarding the validity of a certain payload, this is detected and the beacon node is protected from publishing a block with a potential execution layer consensus bug in it. The BN provides metrics per EL node for the number of successful or failed requests for each type Engine API requests. If an EL node goes offline and connectivity is resoted later, we report the problem and the remedy in edge-triggered fashion. * More progress towards implementing Deneb block production in the VC and comparing the value of blocks produced by the EL and the builder API. * Adds a Makefile target for the zhejiang testnet
2023-03-05 01:40:21 +00:00
proc trackEngineApiRequest(connection: ELConnection,
request: FutureBase, requestName: string,
Don't report an 'EL connection degraded' warning when the EL is not synced (#4761)
2023-03-21 22:19:36 +00:00
startTime: Moment, deadline: Future[void],
failureAllowed = false) =
bump chronos (#5039) * separate futures import / Defect cleanups * unittest2 fix to allow `stackTrace` as a symbol name generally
2023-06-08 13:42:19 +00:00
request.addCallback do (udata: pointer) {.gcsafe, raises: [].}:
More metrics for the EL monitor (#4707) * `engine_api_response_time` provides a histogram for the Engine API response times for each unique pair ot URL and request type. * All engine API requests are now tracked Other changes: The client will no longer exit on start-up if it fails to connect to a properly configured EL node.
2023-03-09 17:29:04 +00:00
# TODO `udata` is nil here. How come?
# This forces us to create a GC cycle between the Future and the closure
if request.completed:
Fix an accidentally wrong partial git stage
2023-03-09 20:28:46 +00:00
engine_api_request_duration_seconds.observe(
More metrics for the EL monitor (#4707) * `engine_api_response_time` provides a histogram for the Engine API response times for each unique pair ot URL and request type. * All engine API requests are now tracked Other changes: The client will no longer exit on start-up if it fails to connect to a properly configured EL node.
2023-03-09 17:29:04 +00:00
float(milliseconds(Moment.now - startTime)) / 1000.0,
[connection.engineUrl.url, requestName])
connection.setWorkingState()
bump chronos (#5039) * separate futures import / Defect cleanups * unittest2 fix to allow `stackTrace` as a symbol name generally
2023-06-08 13:42:19 +00:00
deadline.addCallback do (udata: pointer) {.gcsafe, raises: [].}:
Support for driving multiple EL nodes from a single Nimbus BN (#4465) * Support for driving multiple EL nodes from a single Nimbus BN Full list of changes: * Eth1Monitor has been renamed to ELManager to match its current responsibilities better. * The ELManager is no longer optional in the code (it won't have a nil value under any circumstances). * The support for subscribing for headers was removed as it only worked with WebSockets and contributed significant complexity while bringing only a very minor advantage. * The `--web3-url` parameter has been deprecated in favor of a new `--el` parameter. The new parameter has a reasonable default value and supports specifying a different JWT for each connection. Each connection can also be configured with a different set of responsibilities (e.g. download deposits, validate blocks and/or produce blocks). On the command-line, these properties can be configured through URL properties stored in the #anchor part of the URL. In TOML files, they come with a very natural syntax (althrough the URL scheme is also supported). * The previously scattered EL-related state and logic is now moved to `eth1_monitor.nim` (this module will be renamed to `el_manager.nim` in a follow-up commit). State is assigned properly either to the `ELManager` or the to individual `ELConnection` objects where appropriate. The ELManager executes all Engine API requests against all attached EL nodes, in parallel. It compares their results and if there is a disagreement regarding the validity of a certain payload, this is detected and the beacon node is protected from publishing a block with a potential execution layer consensus bug in it. The BN provides metrics per EL node for the number of successful or failed requests for each type Engine API requests. If an EL node goes offline and connectivity is resoted later, we report the problem and the remedy in edge-triggered fashion. * More progress towards implementing Deneb block production in the VC and comparing the value of blocks produced by the EL and the builder API. * Adds a Makefile target for the zhejiang testnet
2023-03-05 01:40:21 +00:00
if not request.finished:
BN+LC+SN: Fix cancellation deprecate warnings. (#5455) * Fix deprecation cancel() warnings in SN, BN, LC. * Fix missing import.
2023-09-22 11:06:27 +00:00
request.cancelSoon()
Support for driving multiple EL nodes from a single Nimbus BN (#4465) * Support for driving multiple EL nodes from a single Nimbus BN Full list of changes: * Eth1Monitor has been renamed to ELManager to match its current responsibilities better. * The ELManager is no longer optional in the code (it won't have a nil value under any circumstances). * The support for subscribing for headers was removed as it only worked with WebSockets and contributed significant complexity while bringing only a very minor advantage. * The `--web3-url` parameter has been deprecated in favor of a new `--el` parameter. The new parameter has a reasonable default value and supports specifying a different JWT for each connection. Each connection can also be configured with a different set of responsibilities (e.g. download deposits, validate blocks and/or produce blocks). On the command-line, these properties can be configured through URL properties stored in the #anchor part of the URL. In TOML files, they come with a very natural syntax (althrough the URL scheme is also supported). * The previously scattered EL-related state and logic is now moved to `eth1_monitor.nim` (this module will be renamed to `el_manager.nim` in a follow-up commit). State is assigned properly either to the `ELManager` or the to individual `ELConnection` objects where appropriate. The ELManager executes all Engine API requests against all attached EL nodes, in parallel. It compares their results and if there is a disagreement regarding the validity of a certain payload, this is detected and the beacon node is protected from publishing a block with a potential execution layer consensus bug in it. The BN provides metrics per EL node for the number of successful or failed requests for each type Engine API requests. If an EL node goes offline and connectivity is resoted later, we report the problem and the remedy in edge-triggered fashion. * More progress towards implementing Deneb block production in the VC and comparing the value of blocks produced by the EL and the builder API. * Adds a Makefile target for the zhejiang testnet
2023-03-05 01:40:21 +00:00
engine_api_timeouts.inc(1, [connection.engineUrl.url, requestName])
Add logging for EL request time outs
2023-05-02 13:52:26 +00:00
if not failureAllowed:
connection.setDegradedState(requestName, 0, "Request timed out")
Support for driving multiple EL nodes from a single Nimbus BN (#4465) * Support for driving multiple EL nodes from a single Nimbus BN Full list of changes: * Eth1Monitor has been renamed to ELManager to match its current responsibilities better. * The ELManager is no longer optional in the code (it won't have a nil value under any circumstances). * The support for subscribing for headers was removed as it only worked with WebSockets and contributed significant complexity while bringing only a very minor advantage. * The `--web3-url` parameter has been deprecated in favor of a new `--el` parameter. The new parameter has a reasonable default value and supports specifying a different JWT for each connection. Each connection can also be configured with a different set of responsibilities (e.g. download deposits, validate blocks and/or produce blocks). On the command-line, these properties can be configured through URL properties stored in the #anchor part of the URL. In TOML files, they come with a very natural syntax (althrough the URL scheme is also supported). * The previously scattered EL-related state and logic is now moved to `eth1_monitor.nim` (this module will be renamed to `el_manager.nim` in a follow-up commit). State is assigned properly either to the `ELManager` or the to individual `ELConnection` objects where appropriate. The ELManager executes all Engine API requests against all attached EL nodes, in parallel. It compares their results and if there is a disagreement regarding the validity of a certain payload, this is detected and the beacon node is protected from publishing a block with a potential execution layer consensus bug in it. The BN provides metrics per EL node for the number of successful or failed requests for each type Engine API requests. If an EL node goes offline and connectivity is resoted later, we report the problem and the remedy in edge-triggered fashion. * More progress towards implementing Deneb block production in the VC and comparing the value of blocks produced by the EL and the builder API. * Adds a Makefile target for the zhejiang testnet
2023-03-05 01:40:21 +00:00
else:
let statusCode = if not request.failed:
200
elif request.error of ErrorResponse:
((ref ErrorResponse) request.error).status
else:
0
Don't report an 'EL connection degraded' warning when the EL is not synced (#4761)
2023-03-21 22:19:36 +00:00
if request.failed and not failureAllowed:
More metrics for the EL monitor (#4707) * `engine_api_response_time` provides a histogram for the Engine API response times for each unique pair ot URL and request type. * All engine API requests are now tracked Other changes: The client will no longer exit on start-up if it fails to connect to a properly configured EL node.
2023-03-09 17:29:04 +00:00
connection.setDegradedState(requestName, statusCode, request.error.msg)
Support for driving multiple EL nodes from a single Nimbus BN (#4465) * Support for driving multiple EL nodes from a single Nimbus BN Full list of changes: * Eth1Monitor has been renamed to ELManager to match its current responsibilities better. * The ELManager is no longer optional in the code (it won't have a nil value under any circumstances). * The support for subscribing for headers was removed as it only worked with WebSockets and contributed significant complexity while bringing only a very minor advantage. * The `--web3-url` parameter has been deprecated in favor of a new `--el` parameter. The new parameter has a reasonable default value and supports specifying a different JWT for each connection. Each connection can also be configured with a different set of responsibilities (e.g. download deposits, validate blocks and/or produce blocks). On the command-line, these properties can be configured through URL properties stored in the #anchor part of the URL. In TOML files, they come with a very natural syntax (althrough the URL scheme is also supported). * The previously scattered EL-related state and logic is now moved to `eth1_monitor.nim` (this module will be renamed to `el_manager.nim` in a follow-up commit). State is assigned properly either to the `ELManager` or the to individual `ELConnection` objects where appropriate. The ELManager executes all Engine API requests against all attached EL nodes, in parallel. It compares their results and if there is a disagreement regarding the validity of a certain payload, this is detected and the beacon node is protected from publishing a block with a potential execution layer consensus bug in it. The BN provides metrics per EL node for the number of successful or failed requests for each type Engine API requests. If an EL node goes offline and connectivity is resoted later, we report the problem and the remedy in edge-triggered fashion. * More progress towards implementing Deneb block production in the VC and comparing the value of blocks produced by the EL and the builder API. * Adds a Makefile target for the zhejiang testnet
2023-03-05 01:40:21 +00:00
engine_api_responses.inc(1, [connection.engineUrl.url, requestName, $statusCode])
template awaitOrRaiseOnTimeout[T](fut: Future[T],
timeout: Duration): T =
awaitWithTimeout(fut, timeout):
raise newException(DataProviderTimeout, "Timeout")
More metrics for the EL monitor (#4707) * `engine_api_response_time` provides a histogram for the Engine API response times for each unique pair ot URL and request type. * All engine API requests are now tracked Other changes: The client will no longer exit on start-up if it fails to connect to a properly configured EL node.
2023-03-09 17:29:04 +00:00
template trackedRequestWithTimeout[T](connection: ELConnection,
requestName: static string,
lazyRequestExpression: Future[T],
Don't report an 'EL connection degraded' warning when the EL is not synced (#4761)
2023-03-21 22:19:36 +00:00
timeout: Duration,
failureAllowed = false): T =
More metrics for the EL monitor (#4707) * `engine_api_response_time` provides a histogram for the Engine API response times for each unique pair ot URL and request type. * All engine API requests are now tracked Other changes: The client will no longer exit on start-up if it fails to connect to a properly configured EL node.
2023-03-09 17:29:04 +00:00
let
connectionParam = connection
startTime = Moment.now
deadline = sleepAsync(timeout)
request = lazyRequestExpression
Don't report an 'EL connection degraded' warning when the EL is not synced (#4761)
2023-03-21 22:19:36 +00:00
connectionParam.trackEngineApiRequest(
request, requestName, startTime, deadline, failureAllowed)
More metrics for the EL monitor (#4707) * `engine_api_response_time` provides a histogram for the Engine API response times for each unique pair ot URL and request type. * All engine API requests are now tracked Other changes: The client will no longer exit on start-up if it fails to connect to a properly configured EL node.
2023-03-09 17:29:04 +00:00
awaitWithTimeout(request, deadline):
raise newException(DataProviderTimeout, "Timeout")
Support for driving multiple EL nodes from a single Nimbus BN (#4465) * Support for driving multiple EL nodes from a single Nimbus BN Full list of changes: * Eth1Monitor has been renamed to ELManager to match its current responsibilities better. * The ELManager is no longer optional in the code (it won't have a nil value under any circumstances). * The support for subscribing for headers was removed as it only worked with WebSockets and contributed significant complexity while bringing only a very minor advantage. * The `--web3-url` parameter has been deprecated in favor of a new `--el` parameter. The new parameter has a reasonable default value and supports specifying a different JWT for each connection. Each connection can also be configured with a different set of responsibilities (e.g. download deposits, validate blocks and/or produce blocks). On the command-line, these properties can be configured through URL properties stored in the #anchor part of the URL. In TOML files, they come with a very natural syntax (althrough the URL scheme is also supported). * The previously scattered EL-related state and logic is now moved to `eth1_monitor.nim` (this module will be renamed to `el_manager.nim` in a follow-up commit). State is assigned properly either to the `ELManager` or the to individual `ELConnection` objects where appropriate. The ELManager executes all Engine API requests against all attached EL nodes, in parallel. It compares their results and if there is a disagreement regarding the validity of a certain payload, this is detected and the beacon node is protected from publishing a block with a potential execution layer consensus bug in it. The BN provides metrics per EL node for the number of successful or failed requests for each type Engine API requests. If an EL node goes offline and connectivity is resoted later, we report the problem and the remedy in edge-triggered fashion. * More progress towards implementing Deneb block production in the VC and comparing the value of blocks produced by the EL and the builder API. * Adds a Makefile target for the zhejiang testnet
2023-03-05 01:40:21 +00:00
template cfg(m: ELManager): auto =
m.eth1Chain.cfg
template db(m: ELManager): BeaconChainDB =
m.eth1Chain.db
Restore the build support of the -d:has_genesis_detection feature
2021-12-23 14:58:54 +00:00
Support for driving multiple EL nodes from a single Nimbus BN (#4465) * Support for driving multiple EL nodes from a single Nimbus BN Full list of changes: * Eth1Monitor has been renamed to ELManager to match its current responsibilities better. * The ELManager is no longer optional in the code (it won't have a nil value under any circumstances). * The support for subscribing for headers was removed as it only worked with WebSockets and contributed significant complexity while bringing only a very minor advantage. * The `--web3-url` parameter has been deprecated in favor of a new `--el` parameter. The new parameter has a reasonable default value and supports specifying a different JWT for each connection. Each connection can also be configured with a different set of responsibilities (e.g. download deposits, validate blocks and/or produce blocks). On the command-line, these properties can be configured through URL properties stored in the #anchor part of the URL. In TOML files, they come with a very natural syntax (althrough the URL scheme is also supported). * The previously scattered EL-related state and logic is now moved to `eth1_monitor.nim` (this module will be renamed to `el_manager.nim` in a follow-up commit). State is assigned properly either to the `ELManager` or the to individual `ELConnection` objects where appropriate. The ELManager executes all Engine API requests against all attached EL nodes, in parallel. It compares their results and if there is a disagreement regarding the validity of a certain payload, this is detected and the beacon node is protected from publishing a block with a potential execution layer consensus bug in it. The BN provides metrics per EL node for the number of successful or failed requests for each type Engine API requests. If an EL node goes offline and connectivity is resoted later, we report the problem and the remedy in edge-triggered fashion. * More progress towards implementing Deneb block production in the VC and comparing the value of blocks produced by the EL and the builder API. * Adds a Makefile target for the zhejiang testnet
2023-03-05 01:40:21 +00:00
func hasJwtSecret*(m: ELManager): bool =
for c in m.elConnections:
if c.engineUrl.jwtSecret.isSome:
return true
Tentative fix for the Pyrmont Eth1 voting issue
2020-11-19 17:19:03 +00:00
Support for driving multiple EL nodes from a single Nimbus BN (#4465) * Support for driving multiple EL nodes from a single Nimbus BN Full list of changes: * Eth1Monitor has been renamed to ELManager to match its current responsibilities better. * The ELManager is no longer optional in the code (it won't have a nil value under any circumstances). * The support for subscribing for headers was removed as it only worked with WebSockets and contributed significant complexity while bringing only a very minor advantage. * The `--web3-url` parameter has been deprecated in favor of a new `--el` parameter. The new parameter has a reasonable default value and supports specifying a different JWT for each connection. Each connection can also be configured with a different set of responsibilities (e.g. download deposits, validate blocks and/or produce blocks). On the command-line, these properties can be configured through URL properties stored in the #anchor part of the URL. In TOML files, they come with a very natural syntax (althrough the URL scheme is also supported). * The previously scattered EL-related state and logic is now moved to `eth1_monitor.nim` (this module will be renamed to `el_manager.nim` in a follow-up commit). State is assigned properly either to the `ELManager` or the to individual `ELConnection` objects where appropriate. The ELManager executes all Engine API requests against all attached EL nodes, in parallel. It compares their results and if there is a disagreement regarding the validity of a certain payload, this is detected and the beacon node is protected from publishing a block with a potential execution layer consensus bug in it. The BN provides metrics per EL node for the number of successful or failed requests for each type Engine API requests. If an EL node goes offline and connectivity is resoted later, we report the problem and the remedy in edge-triggered fashion. * More progress towards implementing Deneb block production in the VC and comparing the value of blocks produced by the EL and the builder API. * Adds a Makefile target for the zhejiang testnet
2023-03-05 01:40:21 +00:00
func isSynced*(m: ELManager): bool =
m.syncTargetBlock.isSome and
m.eth1Chain.blocks.len > 0 and
m.syncTargetBlock.get <= m.eth1Chain.blocks[^1].number
Add Eth1 deposits simulation to block_sim
2020-12-03 04:30:35 +00:00
Support for driving multiple EL nodes from a single Nimbus BN (#4465) * Support for driving multiple EL nodes from a single Nimbus BN Full list of changes: * Eth1Monitor has been renamed to ELManager to match its current responsibilities better. * The ELManager is no longer optional in the code (it won't have a nil value under any circumstances). * The support for subscribing for headers was removed as it only worked with WebSockets and contributed significant complexity while bringing only a very minor advantage. * The `--web3-url` parameter has been deprecated in favor of a new `--el` parameter. The new parameter has a reasonable default value and supports specifying a different JWT for each connection. Each connection can also be configured with a different set of responsibilities (e.g. download deposits, validate blocks and/or produce blocks). On the command-line, these properties can be configured through URL properties stored in the #anchor part of the URL. In TOML files, they come with a very natural syntax (althrough the URL scheme is also supported). * The previously scattered EL-related state and logic is now moved to `eth1_monitor.nim` (this module will be renamed to `el_manager.nim` in a follow-up commit). State is assigned properly either to the `ELManager` or the to individual `ELConnection` objects where appropriate. The ELManager executes all Engine API requests against all attached EL nodes, in parallel. It compares their results and if there is a disagreement regarding the validity of a certain payload, this is detected and the beacon node is protected from publishing a block with a potential execution layer consensus bug in it. The BN provides metrics per EL node for the number of successful or failed requests for each type Engine API requests. If an EL node goes offline and connectivity is resoted later, we report the problem and the remedy in edge-triggered fashion. * More progress towards implementing Deneb block production in the VC and comparing the value of blocks produced by the EL and the builder API. * Adds a Makefile target for the zhejiang testnet
2023-03-05 01:40:21 +00:00
template eth1ChainBlocks*(m: ELManager): Deque[Eth1Block] =
m.eth1Chain.blocks
fix obtaining deposits after connection loss (#3943) * fix obtaining deposits after connection loss When an error occurs during Eth1 deposits import, the already imported blocks are kept while the connection to the EL is re-established. However, the corresponding merkleizer is not persisted, leading to any future deposits no longer being properly imported. This is quite common when syncing a fresh Nimbus instance against an already-synced Geth EL. Fixed by persisting the head merkleizer together with the blocks.
2022-08-09 21:32:34 +00:00
Support for driving multiple EL nodes from a single Nimbus BN (#4465) * Support for driving multiple EL nodes from a single Nimbus BN Full list of changes: * Eth1Monitor has been renamed to ELManager to match its current responsibilities better. * The ELManager is no longer optional in the code (it won't have a nil value under any circumstances). * The support for subscribing for headers was removed as it only worked with WebSockets and contributed significant complexity while bringing only a very minor advantage. * The `--web3-url` parameter has been deprecated in favor of a new `--el` parameter. The new parameter has a reasonable default value and supports specifying a different JWT for each connection. Each connection can also be configured with a different set of responsibilities (e.g. download deposits, validate blocks and/or produce blocks). On the command-line, these properties can be configured through URL properties stored in the #anchor part of the URL. In TOML files, they come with a very natural syntax (althrough the URL scheme is also supported). * The previously scattered EL-related state and logic is now moved to `eth1_monitor.nim` (this module will be renamed to `el_manager.nim` in a follow-up commit). State is assigned properly either to the `ELManager` or the to individual `ELConnection` objects where appropriate. The ELManager executes all Engine API requests against all attached EL nodes, in parallel. It compares their results and if there is a disagreement regarding the validity of a certain payload, this is detected and the beacon node is protected from publishing a block with a potential execution layer consensus bug in it. The BN provides metrics per EL node for the number of successful or failed requests for each type Engine API requests. If an EL node goes offline and connectivity is resoted later, we report the problem and the remedy in edge-triggered fashion. * More progress towards implementing Deneb block production in the VC and comparing the value of blocks produced by the EL and the builder API. * Adds a Makefile target for the zhejiang testnet
2023-03-05 01:40:21 +00:00
template finalizedDepositsMerkleizer(m: ELManager): auto =
m.eth1Chain.finalizedDepositsMerkleizer
template headMerkleizer(m: ELManager): auto =
m.eth1Chain.headMerkleizer
Fix #1784; Handle Infura HTTPS URLs
2020-11-05 23:11:06 +00:00
Add some useful Eth1 monitor metrics
2020-12-09 22:44:59 +00:00
template toGaugeValue(x: Quantity): int64 =
toGaugeValue(distinctBase x)
Implement split preset/config support (#2710) * Implement split preset/config support This is the initial bulk refactor to introduce runtime config values in a number of places, somewhat replacing the existing mechanism of loading network metadata. It still needs more work, this is the initial refactor that introduces runtime configuration in some of the places that need it. The PR changes the way presets and constants work, to match the spec. In particular, a "preset" now refers to the compile-time configuration while a "cfg" or "RuntimeConfig" is the dynamic part. A single binary can support either mainnet or minimal, but not both. Support for other presets has been removed completely (can be readded, in case there's need). There's a number of outstanding tasks: * `SECONDS_PER_SLOT` still needs fixing * loading custom runtime configs needs redoing * checking constants against YAML file * yeerongpilly support `build/nimbus_beacon_node --network=yeerongpilly --discv5:no --log-level=DEBUG` * load fork epoch from config * fix fork digest sent in status * nicer error string for request failures * fix tools * one more * fixup * fixup * fixup * use "standard" network definition folder in local testnet Files are loaded from their standard locations, including genesis etc, to conform to the format used in the `eth2-networks` repo. * fix launch scripts, allow unknown config values * fix base config of rest test * cleanups * bundle mainnet config using common loader * fix spec links and names * only include supported preset in binary * drop yeerongpilly, add altair-devnet-0, support boot_enr.yaml
2021-07-12 13:01:38 +00:00
# TODO: Add cfg validation
Working test suite with run-time presets
2020-07-07 23:02:14 +00:00
# MIN_GENESIS_ACTIVE_VALIDATOR_COUNT should be larger than SLOTS_PER_EPOCH
Implement split preset/config support (#2710) * Implement split preset/config support This is the initial bulk refactor to introduce runtime config values in a number of places, somewhat replacing the existing mechanism of loading network metadata. It still needs more work, this is the initial refactor that introduces runtime configuration in some of the places that need it. The PR changes the way presets and constants work, to match the spec. In particular, a "preset" now refers to the compile-time configuration while a "cfg" or "RuntimeConfig" is the dynamic part. A single binary can support either mainnet or minimal, but not both. Support for other presets has been removed completely (can be readded, in case there's need). There's a number of outstanding tasks: * `SECONDS_PER_SLOT` still needs fixing * loading custom runtime configs needs redoing * checking constants against YAML file * yeerongpilly support `build/nimbus_beacon_node --network=yeerongpilly --discv5:no --log-level=DEBUG` * load fork epoch from config * fix fork digest sent in status * nicer error string for request failures * fix tools * one more * fixup * fixup * fixup * use "standard" network definition folder in local testnet Files are loaded from their standard locations, including genesis etc, to conform to the format used in the `eth2-networks` repo. * fix launch scripts, allow unknown config values * fix base config of rest test * cleanups * bundle mainnet config using common loader * fix spec links and names * only include supported preset in binary * drop yeerongpilly, add altair-devnet-0, support boot_enr.yaml
2021-07-12 13:01:38 +00:00
# doAssert SECONDS_PER_ETH1_BLOCK * cfg.ETH1_FOLLOW_DISTANCE < GENESIS_DELAY,
Working test suite with run-time presets
2020-07-07 23:02:14 +00:00
# "Invalid configuration: GENESIS_DELAY is set too low"
Spec-compliant implementation of Eth1 monitoring; Eth1-enabled local sim BEWARE! This commit will trigger a stack overflow during local sim
2020-03-24 11:13:07 +00:00
automated consensus spec URL updating to v1.4.0-beta.1 (#5280)
2023-08-09 03:58:47 +00:00
# https://github.com/ethereum/consensus-specs/blob/v1.4.0-beta.1/specs/phase0/validator.md#get_eth1_data
use ForkedHashedBeaconState in StateData (#2634) * use ForkedHashedBeaconState in StateData * fix FAR_FUTURE_EPOCH -> slot overflow; almost always use assign() * avoid stack allocation in maybeUpgradeStateToAltair() * create and use dispatch functions for check_attester_slashing(), check_proposer_slashing(), and check_voluntary_exit() * use getStateRoot() instead of various state.data.hbsPhase0.root * remove withStateVars.hashedState(), which doesn't work as a design anymore * introduce spec/datatypes/altair into beacon_chain_db * fix inefficient codegen for getStateField(largeStateField) * state_transition_slots() doesn't either need/use blocks or runtime presets * combine process_slots(HBS)/state_transition_slots(HBS) which differ only in last-slot htr optimization * getStateField(StateData, ...) was replaced by getStateField(ForkedHashedBeaconState, ...) * fix rollback * switch some state_transition(), process_slots, makeTestBlocks(), etc to use ForkedHashedBeaconState * remove state_transition(phase0.HashedBeaconState) * remove process_slots(phase0.HashedBeaconState) * remove state_transition_block(phase0.HashedBeaconState) * remove unused callWithBS(); separate case expression from if statement * switch back from nested-ref-object construction to (ref Foo)(Bar())
2021-06-11 17:51:46 +00:00
func compute_time_at_slot(genesis_time: uint64, slot: Slot): uint64 =
genesis_time + slot * SECONDS_PER_SLOT
Spec-compliant implementation of Eth1 monitoring; Eth1-enabled local sim BEWARE! This commit will trigger a stack overflow during local sim
2020-03-24 11:13:07 +00:00
automated consensus spec URL updating to v1.4.0-beta.1 (#5280)
2023-08-09 03:58:47 +00:00
# https://github.com/ethereum/consensus-specs/blob/v1.4.0-beta.1/specs/phase0/validator.md#get_eth1_data
cleanups, partly from kintsugi branch (#3161) * cleanups, partly from kintsugi branch * re-export shortLog(EthBlock) and preserve exception messages in batchVerify and processBatch
2021-12-05 17:32:41 +00:00
func voting_period_start_time(state: ForkedHashedBeaconState): uint64 =
switch 12 unchanged spec references from 0.11.x to 0.12.1; line-wrap and remove pointless-to-counterproductive "return"
2020-06-15 09:38:05 +00:00
let eth1_voting_period_start_slot =
use StateData in place of BeaconState outside state transition code (#2551) * use StateData in place of BeaconState outside state transition code * propagate more StateData usage * remove withStateVars().state * wrap get_beacon_committee(BeaconState, ...) as gbc(StateData, ...) * switch makeAttestation() to use StateData * use StateData wrapper/dispatcher for get_committee_count_per_slot() * convert AttestationCache.init(), weak subjectivity functions, and updateValidatorMetrics() * add get_shuffled_active_validator_indices(StateData) and get_block_root_at_slot(StateData) * switch makeAttestationData() to StateData * sync AllTests-mainnet.md after rebase
2021-05-21 09:23:28 +00:00
getStateField(state, slot) - getStateField(state, slot) mod
SLOTS_PER_ETH1_VOTING_PERIOD.uint64
use ForkedHashedBeaconState in StateData (#2634) * use ForkedHashedBeaconState in StateData * fix FAR_FUTURE_EPOCH -> slot overflow; almost always use assign() * avoid stack allocation in maybeUpgradeStateToAltair() * create and use dispatch functions for check_attester_slashing(), check_proposer_slashing(), and check_voluntary_exit() * use getStateRoot() instead of various state.data.hbsPhase0.root * remove withStateVars.hashedState(), which doesn't work as a design anymore * introduce spec/datatypes/altair into beacon_chain_db * fix inefficient codegen for getStateField(largeStateField) * state_transition_slots() doesn't either need/use blocks or runtime presets * combine process_slots(HBS)/state_transition_slots(HBS) which differ only in last-slot htr optimization * getStateField(StateData, ...) was replaced by getStateField(ForkedHashedBeaconState, ...) * fix rollback * switch some state_transition(), process_slots, makeTestBlocks(), etc to use ForkedHashedBeaconState * remove state_transition(phase0.HashedBeaconState) * remove process_slots(phase0.HashedBeaconState) * remove state_transition_block(phase0.HashedBeaconState) * remove unused callWithBS(); separate case expression from if statement * switch back from nested-ref-object construction to (ref Foo)(Bar())
2021-06-11 17:51:46 +00:00
compute_time_at_slot(
getStateField(state, genesis_time), eth1_voting_period_start_slot)
Spec-compliant implementation of Eth1 monitoring; Eth1-enabled local sim BEWARE! This commit will trigger a stack overflow during local sim
2020-03-24 11:13:07 +00:00
automated consensus spec URL updating to v1.4.0-beta.1 (#5280)
2023-08-09 03:58:47 +00:00
# https://github.com/ethereum/consensus-specs/blob/v1.4.0-beta.1/specs/phase0/validator.md#get_eth1_data
Implement split preset/config support (#2710) * Implement split preset/config support This is the initial bulk refactor to introduce runtime config values in a number of places, somewhat replacing the existing mechanism of loading network metadata. It still needs more work, this is the initial refactor that introduces runtime configuration in some of the places that need it. The PR changes the way presets and constants work, to match the spec. In particular, a "preset" now refers to the compile-time configuration while a "cfg" or "RuntimeConfig" is the dynamic part. A single binary can support either mainnet or minimal, but not both. Support for other presets has been removed completely (can be readded, in case there's need). There's a number of outstanding tasks: * `SECONDS_PER_SLOT` still needs fixing * loading custom runtime configs needs redoing * checking constants against YAML file * yeerongpilly support `build/nimbus_beacon_node --network=yeerongpilly --discv5:no --log-level=DEBUG` * load fork epoch from config * fix fork digest sent in status * nicer error string for request failures * fix tools * one more * fixup * fixup * fixup * use "standard" network definition folder in local testnet Files are loaded from their standard locations, including genesis etc, to conform to the format used in the `eth2-networks` repo. * fix launch scripts, allow unknown config values * fix base config of rest test * cleanups * bundle mainnet config using common loader * fix spec links and names * only include supported preset in binary * drop yeerongpilly, add altair-devnet-0, support boot_enr.yaml
2021-07-12 13:01:38 +00:00
func is_candidate_block(cfg: RuntimeConfig,
Add more eth1-related logging
2020-11-20 14:05:37 +00:00
blk: Eth1Block,
period_start: uint64): bool =
Implement split preset/config support (#2710) * Implement split preset/config support This is the initial bulk refactor to introduce runtime config values in a number of places, somewhat replacing the existing mechanism of loading network metadata. It still needs more work, this is the initial refactor that introduces runtime configuration in some of the places that need it. The PR changes the way presets and constants work, to match the spec. In particular, a "preset" now refers to the compile-time configuration while a "cfg" or "RuntimeConfig" is the dynamic part. A single binary can support either mainnet or minimal, but not both. Support for other presets has been removed completely (can be readded, in case there's need). There's a number of outstanding tasks: * `SECONDS_PER_SLOT` still needs fixing * loading custom runtime configs needs redoing * checking constants against YAML file * yeerongpilly support `build/nimbus_beacon_node --network=yeerongpilly --discv5:no --log-level=DEBUG` * load fork epoch from config * fix fork digest sent in status * nicer error string for request failures * fix tools * one more * fixup * fixup * fixup * use "standard" network definition folder in local testnet Files are loaded from their standard locations, including genesis etc, to conform to the format used in the `eth2-networks` repo. * fix launch scripts, allow unknown config values * fix base config of rest test * cleanups * bundle mainnet config using common loader * fix spec links and names * only include supported preset in binary * drop yeerongpilly, add altair-devnet-0, support boot_enr.yaml
2021-07-12 13:01:38 +00:00
(blk.timestamp + cfg.SECONDS_PER_ETH1_BLOCK * cfg.ETH1_FOLLOW_DISTANCE <= period_start) and
(blk.timestamp + cfg.SECONDS_PER_ETH1_BLOCK * cfg.ETH1_FOLLOW_DISTANCE * 2 >= period_start)
Spec-compliant implementation of Eth1 monitoring; Eth1-enabled local sim BEWARE! This commit will trigger a stack overflow during local sim
2020-03-24 11:13:07 +00:00
Smooth out the differences between Ganache and Infura; Working local sim and Altona target
2020-06-19 17:42:28 +00:00
func asEth2Digest*(x: BlockHash): Eth2Digest =
Spec-compliant implementation of Eth1 monitoring; Eth1-enabled local sim BEWARE! This commit will trigger a stack overflow during local sim
2020-03-24 11:13:07 +00:00
Eth2Digest(data: array[32, byte](x))
kiln beacon node (#3540) * kiln bn * use version of beacon_chain_db * have Eth1Monitor abstract more tightly over web3provider
2022-03-25 11:40:10 +00:00
template asBlockHash*(x: Eth2Digest): BlockHash =
Spec-compliant implementation of Eth1 monitoring; Eth1-enabled local sim BEWARE! This commit will trigger a stack overflow during local sim
2020-03-24 11:13:07 +00:00
BlockHash(x.data)
capella forkchoiceUpdated support (#4462) * capella forkchoiceUpdated support * match V2 fcU with V2 getPayload
2023-01-06 21:01:10 +00:00
func asConsensusWithdrawal(w: WithdrawalV1): capella.Withdrawal =
capella.Withdrawal(
index: w.index.uint64,
validator_index: w.validatorIndex.uint64,
address: ExecutionAddress(data: w.address.distinctBase),
update some beacon API spec URLs; fix some Name and DuplicateModuleImport hints (#4929)
2023-05-10 10:20:55 +00:00
amount: Gwei w.amount)
capella forkchoiceUpdated support (#4462) * capella forkchoiceUpdated support * match V2 fcU with V2 getPayload
2023-01-06 21:01:10 +00:00
func asEngineWithdrawal(w: capella.Withdrawal): WithdrawalV1 =
WithdrawalV1(
index: Quantity(w.index),
validatorIndex: Quantity(w.validator_index),
address: Address(w.address.data),
execution/engine withdrawals amount in uint64 gwei (#4509)
2023-01-14 17:26:57 +00:00
amount: Quantity(w.amount))
capella forkchoiceUpdated support (#4462) * capella forkchoiceUpdated support * match V2 fcU with V2 getPayload
2023-01-06 21:01:10 +00:00
Support for driving multiple EL nodes from a single Nimbus BN (#4465) * Support for driving multiple EL nodes from a single Nimbus BN Full list of changes: * Eth1Monitor has been renamed to ELManager to match its current responsibilities better. * The ELManager is no longer optional in the code (it won't have a nil value under any circumstances). * The support for subscribing for headers was removed as it only worked with WebSockets and contributed significant complexity while bringing only a very minor advantage. * The `--web3-url` parameter has been deprecated in favor of a new `--el` parameter. The new parameter has a reasonable default value and supports specifying a different JWT for each connection. Each connection can also be configured with a different set of responsibilities (e.g. download deposits, validate blocks and/or produce blocks). On the command-line, these properties can be configured through URL properties stored in the #anchor part of the URL. In TOML files, they come with a very natural syntax (althrough the URL scheme is also supported). * The previously scattered EL-related state and logic is now moved to `eth1_monitor.nim` (this module will be renamed to `el_manager.nim` in a follow-up commit). State is assigned properly either to the `ELManager` or the to individual `ELConnection` objects where appropriate. The ELManager executes all Engine API requests against all attached EL nodes, in parallel. It compares their results and if there is a disagreement regarding the validity of a certain payload, this is detected and the beacon node is protected from publishing a block with a potential execution layer consensus bug in it. The BN provides metrics per EL node for the number of successful or failed requests for each type Engine API requests. If an EL node goes offline and connectivity is resoted later, we report the problem and the remedy in edge-triggered fashion. * More progress towards implementing Deneb block production in the VC and comparing the value of blocks produced by the EL and the builder API. * Adds a Makefile target for the zhejiang testnet
2023-03-05 01:40:21 +00:00
func asConsensusType*(rpcExecutionPayload: ExecutionPayloadV1):
update 10 modules from using merge to bellatrix (#3272)
2022-01-12 14:50:30 +00:00
bellatrix.ExecutionPayload =
stylecheck fixes (#3595)
2022-04-15 12:46:56 +00:00
template getTransaction(tt: TypedTransaction): bellatrix.Transaction =
bellatrix.Transaction.init(tt.distinctBase)
convert between engine and consensus ExecutionPayloads (#3228) * convert between engine and consensus ExecutionPayloads
2022-01-03 12:22:56 +00:00
update 10 modules from using merge to bellatrix (#3272)
2022-01-12 14:50:30 +00:00
bellatrix.ExecutionPayload(
convert between engine and consensus ExecutionPayloads (#3228) * convert between engine and consensus ExecutionPayloads
2022-01-03 12:22:56 +00:00
parent_hash: rpcExecutionPayload.parentHash.asEth2Digest,
feeRecipient:
ExecutionAddress(data: rpcExecutionPayload.feeRecipient.distinctBase),
state_root: rpcExecutionPayload.stateRoot.asEth2Digest,
receipts_root: rpcExecutionPayload.receiptsRoot.asEth2Digest,
logs_bloom: BloomLogs(data: rpcExecutionPayload.logsBloom.distinctBase),
rename random -> prev_randao in Bellatrix for CL specs v1.1.10 (#3460)
2022-03-03 16:08:14 +00:00
prev_randao: rpcExecutionPayload.prevRandao.asEth2Digest,
convert between engine and consensus ExecutionPayloads (#3228) * convert between engine and consensus ExecutionPayloads
2022-01-03 12:22:56 +00:00
block_number: rpcExecutionPayload.blockNumber.uint64,
gas_limit: rpcExecutionPayload.gasLimit.uint64,
gas_used: rpcExecutionPayload.gasUsed.uint64,
timestamp: rpcExecutionPayload.timestamp.uint64,
Local sim impovements (#4551) * Local sim impovements * Added support for running Capella and EIP-4844 simulations by downloading the correct version of Geth. * Added support for using Nimbus remote signer and Web3Signer. Use 2 out of 3 threshold signing configuration in the mainnet configuration and regular remote signing in the minimal one. * The local testnet simulation can now use a payload builder. This is currently not activated in CI due to lack of automated procedures for installing third-party relays or builders. You are adviced to use mergemock for now, but for most realistic results, we can create a simple builder based on the nimbus-eth1 codebase that will be able to propose transactions from the regular network mempool. * Start the simulation from a merged state. This would allow us to start removing pre-merge functionality such as the gossip subsciption logic. The commit also removes the merge-forcing hack installed after the TTD removal. * Consolidate all the tools used in the local simulation into a single `ncli_testnet` binary.
2023-02-23 02:10:07 +00:00
extra_data: List[byte, MAX_EXTRA_DATA_BYTES].init(rpcExecutionPayload.extraData.bytes),
fix ExecutionPayload(Header) JSON serialization (#3679)
2022-06-01 12:57:28 +00:00
base_fee_per_gas: rpcExecutionPayload.baseFeePerGas,
convert between engine and consensus ExecutionPayloads (#3228) * convert between engine and consensus ExecutionPayloads
2022-01-03 12:22:56 +00:00
block_hash: rpcExecutionPayload.blockHash.asEth2Digest,
update 10 modules from using merge to bellatrix (#3272)
2022-01-12 14:50:30 +00:00
transactions: List[bellatrix.Transaction, MAX_TRANSACTIONS_PER_PAYLOAD].init(
convert between engine and consensus ExecutionPayloads (#3228) * convert between engine and consensus ExecutionPayloads
2022-01-03 12:22:56 +00:00
mapIt(rpcExecutionPayload.transactions, it.getTransaction)))
Support for driving multiple EL nodes from a single Nimbus BN (#4465) * Support for driving multiple EL nodes from a single Nimbus BN Full list of changes: * Eth1Monitor has been renamed to ELManager to match its current responsibilities better. * The ELManager is no longer optional in the code (it won't have a nil value under any circumstances). * The support for subscribing for headers was removed as it only worked with WebSockets and contributed significant complexity while bringing only a very minor advantage. * The `--web3-url` parameter has been deprecated in favor of a new `--el` parameter. The new parameter has a reasonable default value and supports specifying a different JWT for each connection. Each connection can also be configured with a different set of responsibilities (e.g. download deposits, validate blocks and/or produce blocks). On the command-line, these properties can be configured through URL properties stored in the #anchor part of the URL. In TOML files, they come with a very natural syntax (althrough the URL scheme is also supported). * The previously scattered EL-related state and logic is now moved to `eth1_monitor.nim` (this module will be renamed to `el_manager.nim` in a follow-up commit). State is assigned properly either to the `ELManager` or the to individual `ELConnection` objects where appropriate. The ELManager executes all Engine API requests against all attached EL nodes, in parallel. It compares their results and if there is a disagreement regarding the validity of a certain payload, this is detected and the beacon node is protected from publishing a block with a potential execution layer consensus bug in it. The BN provides metrics per EL node for the number of successful or failed requests for each type Engine API requests. If an EL node goes offline and connectivity is resoted later, we report the problem and the remedy in edge-triggered fashion. * More progress towards implementing Deneb block production in the VC and comparing the value of blocks produced by the EL and the builder API. * Adds a Makefile target for the zhejiang testnet
2023-03-05 01:40:21 +00:00
func asConsensusType*(payloadWithValue: BellatrixExecutionPayloadWithValue):
bellatrix.ExecutionPayloadForSigning =
bellatrix.ExecutionPayloadForSigning(
executionPayload: payloadWithValue.executionPayload.asConsensusType,
blockValue: payloadWithValue.blockValue)
template maybeDeref[T](o: Option[T]): T = o.get
template maybeDeref[V](v: V): V = v
func asConsensusType*(rpcExecutionPayload: ExecutionPayloadV1OrV2|ExecutionPayloadV2):
support engine API execution payloads with withdrawals (#4358)
2022-11-29 05:02:16 +00:00
capella.ExecutionPayload =
template getTransaction(tt: TypedTransaction): bellatrix.Transaction =
bellatrix.Transaction.init(tt.distinctBase)
capella.ExecutionPayload(
parent_hash: rpcExecutionPayload.parentHash.asEth2Digest,
feeRecipient:
ExecutionAddress(data: rpcExecutionPayload.feeRecipient.distinctBase),
state_root: rpcExecutionPayload.stateRoot.asEth2Digest,
receipts_root: rpcExecutionPayload.receiptsRoot.asEth2Digest,
logs_bloom: BloomLogs(data: rpcExecutionPayload.logsBloom.distinctBase),
prev_randao: rpcExecutionPayload.prevRandao.asEth2Digest,
block_number: rpcExecutionPayload.blockNumber.uint64,
gas_limit: rpcExecutionPayload.gasLimit.uint64,
gas_used: rpcExecutionPayload.gasUsed.uint64,
timestamp: rpcExecutionPayload.timestamp.uint64,
Local sim impovements (#4551) * Local sim impovements * Added support for running Capella and EIP-4844 simulations by downloading the correct version of Geth. * Added support for using Nimbus remote signer and Web3Signer. Use 2 out of 3 threshold signing configuration in the mainnet configuration and regular remote signing in the minimal one. * The local testnet simulation can now use a payload builder. This is currently not activated in CI due to lack of automated procedures for installing third-party relays or builders. You are adviced to use mergemock for now, but for most realistic results, we can create a simple builder based on the nimbus-eth1 codebase that will be able to propose transactions from the regular network mempool. * Start the simulation from a merged state. This would allow us to start removing pre-merge functionality such as the gossip subsciption logic. The commit also removes the merge-forcing hack installed after the TTD removal. * Consolidate all the tools used in the local simulation into a single `ncli_testnet` binary.
2023-02-23 02:10:07 +00:00
extra_data: List[byte, MAX_EXTRA_DATA_BYTES].init(rpcExecutionPayload.extraData.bytes),
support engine API execution payloads with withdrawals (#4358)
2022-11-29 05:02:16 +00:00
base_fee_per_gas: rpcExecutionPayload.baseFeePerGas,
block_hash: rpcExecutionPayload.blockHash.asEth2Digest,
transactions: List[bellatrix.Transaction, MAX_TRANSACTIONS_PER_PAYLOAD].init(
mapIt(rpcExecutionPayload.transactions, it.getTransaction)),
fix effective merge conflict between #s 4357 and 4358 around Withdrawal symbol ambiguity (#4370)
2022-11-29 11:04:36 +00:00
withdrawals: List[capella.Withdrawal, MAX_WITHDRAWALS_PER_PAYLOAD].init(
Support for driving multiple EL nodes from a single Nimbus BN (#4465) * Support for driving multiple EL nodes from a single Nimbus BN Full list of changes: * Eth1Monitor has been renamed to ELManager to match its current responsibilities better. * The ELManager is no longer optional in the code (it won't have a nil value under any circumstances). * The support for subscribing for headers was removed as it only worked with WebSockets and contributed significant complexity while bringing only a very minor advantage. * The `--web3-url` parameter has been deprecated in favor of a new `--el` parameter. The new parameter has a reasonable default value and supports specifying a different JWT for each connection. Each connection can also be configured with a different set of responsibilities (e.g. download deposits, validate blocks and/or produce blocks). On the command-line, these properties can be configured through URL properties stored in the #anchor part of the URL. In TOML files, they come with a very natural syntax (althrough the URL scheme is also supported). * The previously scattered EL-related state and logic is now moved to `eth1_monitor.nim` (this module will be renamed to `el_manager.nim` in a follow-up commit). State is assigned properly either to the `ELManager` or the to individual `ELConnection` objects where appropriate. The ELManager executes all Engine API requests against all attached EL nodes, in parallel. It compares their results and if there is a disagreement regarding the validity of a certain payload, this is detected and the beacon node is protected from publishing a block with a potential execution layer consensus bug in it. The BN provides metrics per EL node for the number of successful or failed requests for each type Engine API requests. If an EL node goes offline and connectivity is resoted later, we report the problem and the remedy in edge-triggered fashion. * More progress towards implementing Deneb block production in the VC and comparing the value of blocks produced by the EL and the builder API. * Adds a Makefile target for the zhejiang testnet
2023-03-05 01:40:21 +00:00
mapIt(maybeDeref rpcExecutionPayload.withdrawals, it.asConsensusWithdrawal)))
func asConsensusType*(payloadWithValue: engine_api.GetPayloadV2Response):
capella.ExecutionPayloadForSigning =
capella.ExecutionPayloadForSigning(
executionPayload: payloadWithValue.executionPayload.asConsensusType,
blockValue: payloadWithValue.blockValue)
support engine API execution payloads with withdrawals (#4358)
2022-11-29 05:02:16 +00:00
Support for driving multiple EL nodes from a single Nimbus BN (#4465) * Support for driving multiple EL nodes from a single Nimbus BN Full list of changes: * Eth1Monitor has been renamed to ELManager to match its current responsibilities better. * The ELManager is no longer optional in the code (it won't have a nil value under any circumstances). * The support for subscribing for headers was removed as it only worked with WebSockets and contributed significant complexity while bringing only a very minor advantage. * The `--web3-url` parameter has been deprecated in favor of a new `--el` parameter. The new parameter has a reasonable default value and supports specifying a different JWT for each connection. Each connection can also be configured with a different set of responsibilities (e.g. download deposits, validate blocks and/or produce blocks). On the command-line, these properties can be configured through URL properties stored in the #anchor part of the URL. In TOML files, they come with a very natural syntax (althrough the URL scheme is also supported). * The previously scattered EL-related state and logic is now moved to `eth1_monitor.nim` (this module will be renamed to `el_manager.nim` in a follow-up commit). State is assigned properly either to the `ELManager` or the to individual `ELConnection` objects where appropriate. The ELManager executes all Engine API requests against all attached EL nodes, in parallel. It compares their results and if there is a disagreement regarding the validity of a certain payload, this is detected and the beacon node is protected from publishing a block with a potential execution layer consensus bug in it. The BN provides metrics per EL node for the number of successful or failed requests for each type Engine API requests. If an EL node goes offline and connectivity is resoted later, we report the problem and the remedy in edge-triggered fashion. * More progress towards implementing Deneb block production in the VC and comparing the value of blocks produced by the EL and the builder API. * Adds a Makefile target for the zhejiang testnet
2023-03-05 01:40:21 +00:00
func asConsensusType*(rpcExecutionPayload: ExecutionPayloadV3):
more eip4844 -> deneb changes (#4666)
2023-02-25 01:03:34 +00:00
deneb.ExecutionPayload =
Wire up engine_newPayloadV3 (#4482) * Wire up eip4844's newPayloadV3 * Add eip4844 test * Update AllTests-mainnet.md and fix typo
2023-01-11 18:21:19 +00:00
template getTransaction(tt: TypedTransaction): bellatrix.Transaction =
bellatrix.Transaction.init(tt.distinctBase)
more eip4844 -> deneb changes (#4666)
2023-02-25 01:03:34 +00:00
deneb.ExecutionPayload(
Wire up engine_newPayloadV3 (#4482) * Wire up eip4844's newPayloadV3 * Add eip4844 test * Update AllTests-mainnet.md and fix typo
2023-01-11 18:21:19 +00:00
parent_hash: rpcExecutionPayload.parentHash.asEth2Digest,
feeRecipient:
ExecutionAddress(data: rpcExecutionPayload.feeRecipient.distinctBase),
state_root: rpcExecutionPayload.stateRoot.asEth2Digest,
receipts_root: rpcExecutionPayload.receiptsRoot.asEth2Digest,
logs_bloom: BloomLogs(data: rpcExecutionPayload.logsBloom.distinctBase),
prev_randao: rpcExecutionPayload.prevRandao.asEth2Digest,
block_number: rpcExecutionPayload.blockNumber.uint64,
gas_limit: rpcExecutionPayload.gasLimit.uint64,
gas_used: rpcExecutionPayload.gasUsed.uint64,
timestamp: rpcExecutionPayload.timestamp.uint64,
Local sim impovements (#4551) * Local sim impovements * Added support for running Capella and EIP-4844 simulations by downloading the correct version of Geth. * Added support for using Nimbus remote signer and Web3Signer. Use 2 out of 3 threshold signing configuration in the mainnet configuration and regular remote signing in the minimal one. * The local testnet simulation can now use a payload builder. This is currently not activated in CI due to lack of automated procedures for installing third-party relays or builders. You are adviced to use mergemock for now, but for most realistic results, we can create a simple builder based on the nimbus-eth1 codebase that will be able to propose transactions from the regular network mempool. * Start the simulation from a merged state. This would allow us to start removing pre-merge functionality such as the gossip subsciption logic. The commit also removes the merge-forcing hack installed after the TTD removal. * Consolidate all the tools used in the local simulation into a single `ncli_testnet` binary.
2023-02-23 02:10:07 +00:00
extra_data: List[byte, MAX_EXTRA_DATA_BYTES].init(rpcExecutionPayload.extraData.bytes),
Wire up engine_newPayloadV3 (#4482) * Wire up eip4844's newPayloadV3 * Add eip4844 test * Update AllTests-mainnet.md and fix typo
2023-01-11 18:21:19 +00:00
base_fee_per_gas: rpcExecutionPayload.baseFeePerGas,
block_hash: rpcExecutionPayload.blockHash.asEth2Digest,
transactions: List[bellatrix.Transaction, MAX_TRANSACTIONS_PER_PAYLOAD].init(
mapIt(rpcExecutionPayload.transactions, it.getTransaction)),
withdrawals: List[capella.Withdrawal, MAX_WITHDRAWALS_PER_PAYLOAD].init(
`excess_data_gas` after `withdrawals` cleanup (#4900) We already updated the field order in the actual `ExecutionPayload`, but in init code and tests / logs etc we still used the old order. Update those occurrences to also match the field order in the struct. Furthermore, add `excess_data_gas` to last entry in `test_eth1_monitor`.
2023-05-05 23:15:47 +00:00
mapIt(rpcExecutionPayload.withdrawals, it.asConsensusWithdrawal)),
Rename "data gas" to "blob gas" (#5216) * Rename data gas to blob gas * Update vendor/nim-eth and vendor/nim-web3
2023-08-02 22:07:57 +00:00
blob_gas_used: rpcExecutionPayload.blobGasUsed.uint64,
excess_blob_gas: rpcExecutionPayload.excessBlobGas.uint64)
Wire up engine_newPayloadV3 (#4482) * Wire up eip4844's newPayloadV3 * Add eip4844 test * Update AllTests-mainnet.md and fix typo
2023-01-11 18:21:19 +00:00
Update nim-web3 (#4875) Update to new version of nim-web3 incorporating the execution API change that merges getPayloadV3 and getBlobsBundleV1.
2023-04-28 19:20:25 +00:00
func asConsensusType*(payload: engine_api.GetPayloadV3Response):
Support for driving multiple EL nodes from a single Nimbus BN (#4465) * Support for driving multiple EL nodes from a single Nimbus BN Full list of changes: * Eth1Monitor has been renamed to ELManager to match its current responsibilities better. * The ELManager is no longer optional in the code (it won't have a nil value under any circumstances). * The support for subscribing for headers was removed as it only worked with WebSockets and contributed significant complexity while bringing only a very minor advantage. * The `--web3-url` parameter has been deprecated in favor of a new `--el` parameter. The new parameter has a reasonable default value and supports specifying a different JWT for each connection. Each connection can also be configured with a different set of responsibilities (e.g. download deposits, validate blocks and/or produce blocks). On the command-line, these properties can be configured through URL properties stored in the #anchor part of the URL. In TOML files, they come with a very natural syntax (althrough the URL scheme is also supported). * The previously scattered EL-related state and logic is now moved to `eth1_monitor.nim` (this module will be renamed to `el_manager.nim` in a follow-up commit). State is assigned properly either to the `ELManager` or the to individual `ELConnection` objects where appropriate. The ELManager executes all Engine API requests against all attached EL nodes, in parallel. It compares their results and if there is a disagreement regarding the validity of a certain payload, this is detected and the beacon node is protected from publishing a block with a potential execution layer consensus bug in it. The BN provides metrics per EL node for the number of successful or failed requests for each type Engine API requests. If an EL node goes offline and connectivity is resoted later, we report the problem and the remedy in edge-triggered fashion. * More progress towards implementing Deneb block production in the VC and comparing the value of blocks produced by the EL and the builder API. * Adds a Makefile target for the zhejiang testnet
2023-03-05 01:40:21 +00:00
deneb.ExecutionPayloadForSigning =
deneb.ExecutionPayloadForSigning(
Update nim-web3 (#4875) Update to new version of nim-web3 incorporating the execution API change that merges getPayloadV3 and getBlobsBundleV1.
2023-04-28 19:20:25 +00:00
executionPayload: payload.executionPayload.asConsensusType,
blockValue: payload.blockValue,
Support for driving multiple EL nodes from a single Nimbus BN (#4465) * Support for driving multiple EL nodes from a single Nimbus BN Full list of changes: * Eth1Monitor has been renamed to ELManager to match its current responsibilities better. * The ELManager is no longer optional in the code (it won't have a nil value under any circumstances). * The support for subscribing for headers was removed as it only worked with WebSockets and contributed significant complexity while bringing only a very minor advantage. * The `--web3-url` parameter has been deprecated in favor of a new `--el` parameter. The new parameter has a reasonable default value and supports specifying a different JWT for each connection. Each connection can also be configured with a different set of responsibilities (e.g. download deposits, validate blocks and/or produce blocks). On the command-line, these properties can be configured through URL properties stored in the #anchor part of the URL. In TOML files, they come with a very natural syntax (althrough the URL scheme is also supported). * The previously scattered EL-related state and logic is now moved to `eth1_monitor.nim` (this module will be renamed to `el_manager.nim` in a follow-up commit). State is assigned properly either to the `ELManager` or the to individual `ELConnection` objects where appropriate. The ELManager executes all Engine API requests against all attached EL nodes, in parallel. It compares their results and if there is a disagreement regarding the validity of a certain payload, this is detected and the beacon node is protected from publishing a block with a potential execution layer consensus bug in it. The BN provides metrics per EL node for the number of successful or failed requests for each type Engine API requests. If an EL node goes offline and connectivity is resoted later, we report the problem and the remedy in edge-triggered fashion. * More progress towards implementing Deneb block production in the VC and comparing the value of blocks produced by the EL and the builder API. * Adds a Makefile target for the zhejiang testnet
2023-03-05 01:40:21 +00:00
# TODO
# The `mapIt` calls below are necessary only because we use different distinct
# types for KZG commitments and Blobs in the `web3` and the `deneb` spec types.
# Both are defined as `array[N, byte]` under the hood.
`KZG(Commitment(s)?|Proof)` > `Kzg$1` (#4917) Fix style-check by using matching capitalization for `nim-kzg` types.
2023-05-09 19:51:36 +00:00
kzgs: KzgCommitments payload.blobsBundle.commitments.mapIt(it.bytes),
Local validator proposals (#5137) * Implement local validator block proposals * Address review feedback * Fix rebase issue
2023-06-30 07:39:41 +00:00
proofs: payload.blobsBundle.proofs.mapIt(it.bytes),
Update nim-web3 (#4875) Update to new version of nim-web3 incorporating the execution API change that merges getPayloadV3 and getBlobsBundleV1.
2023-04-28 19:20:25 +00:00
blobs: Blobs payload.blobsBundle.blobs.mapIt(it.bytes)
Support for driving multiple EL nodes from a single Nimbus BN (#4465) * Support for driving multiple EL nodes from a single Nimbus BN Full list of changes: * Eth1Monitor has been renamed to ELManager to match its current responsibilities better. * The ELManager is no longer optional in the code (it won't have a nil value under any circumstances). * The support for subscribing for headers was removed as it only worked with WebSockets and contributed significant complexity while bringing only a very minor advantage. * The `--web3-url` parameter has been deprecated in favor of a new `--el` parameter. The new parameter has a reasonable default value and supports specifying a different JWT for each connection. Each connection can also be configured with a different set of responsibilities (e.g. download deposits, validate blocks and/or produce blocks). On the command-line, these properties can be configured through URL properties stored in the #anchor part of the URL. In TOML files, they come with a very natural syntax (althrough the URL scheme is also supported). * The previously scattered EL-related state and logic is now moved to `eth1_monitor.nim` (this module will be renamed to `el_manager.nim` in a follow-up commit). State is assigned properly either to the `ELManager` or the to individual `ELConnection` objects where appropriate. The ELManager executes all Engine API requests against all attached EL nodes, in parallel. It compares their results and if there is a disagreement regarding the validity of a certain payload, this is detected and the beacon node is protected from publishing a block with a potential execution layer consensus bug in it. The BN provides metrics per EL node for the number of successful or failed requests for each type Engine API requests. If an EL node goes offline and connectivity is resoted later, we report the problem and the remedy in edge-triggered fashion. * More progress towards implementing Deneb block production in the VC and comparing the value of blocks produced by the EL and the builder API. * Adds a Makefile target for the zhejiang testnet
2023-03-05 01:40:21 +00:00
)
update 10 modules from using merge to bellatrix (#3272)
2022-01-12 14:50:30 +00:00
func asEngineExecutionPayload*(executionPayload: bellatrix.ExecutionPayload):
convert between engine and consensus ExecutionPayloads (#3228) * convert between engine and consensus ExecutionPayloads
2022-01-03 12:22:56 +00:00
ExecutionPayloadV1 =
stylecheck fixes (#3595)
2022-04-15 12:46:56 +00:00
template getTypedTransaction(tt: bellatrix.Transaction): TypedTransaction =
TypedTransaction(tt.distinctBase)
convert between engine and consensus ExecutionPayloads (#3228) * convert between engine and consensus ExecutionPayloads
2022-01-03 12:22:56 +00:00
engine_api.ExecutionPayloadV1(
parentHash: executionPayload.parent_hash.asBlockHash,
enable `styleCheck:usages` (#3573) Some upstream repos still need fixes, but this gets us close enough that style hints can be enabled by default. In general, "canonical" spellings are preferred even if they violate nep-1 - this applies in particular to spec-related stuff like `genesis_validators_root` which appears throughout the codebase.
2022-04-08 16:22:49 +00:00
feeRecipient: Address(executionPayload.fee_recipient.data),
convert between engine and consensus ExecutionPayloads (#3228) * convert between engine and consensus ExecutionPayloads
2022-01-03 12:22:56 +00:00
stateRoot: executionPayload.state_root.asBlockHash,
receiptsRoot: executionPayload.receipts_root.asBlockHash,
logsBloom:
FixedBytes[BYTES_PER_LOGS_BLOOM](executionPayload.logs_bloom.data),
rename random -> prev_randao in Bellatrix for CL specs v1.1.10 (#3460)
2022-03-03 16:08:14 +00:00
prevRandao: executionPayload.prev_randao.asBlockHash,
convert between engine and consensus ExecutionPayloads (#3228) * convert between engine and consensus ExecutionPayloads
2022-01-03 12:22:56 +00:00
blockNumber: Quantity(executionPayload.block_number),
implement more missing capella functionality (#4344)
2022-11-24 07:53:04 +00:00
gasLimit: Quantity(executionPayload.gas_limit),
gasUsed: Quantity(executionPayload.gas_used),
timestamp: Quantity(executionPayload.timestamp),
Local sim impovements (#4551) * Local sim impovements * Added support for running Capella and EIP-4844 simulations by downloading the correct version of Geth. * Added support for using Nimbus remote signer and Web3Signer. Use 2 out of 3 threshold signing configuration in the mainnet configuration and regular remote signing in the minimal one. * The local testnet simulation can now use a payload builder. This is currently not activated in CI due to lack of automated procedures for installing third-party relays or builders. You are adviced to use mergemock for now, but for most realistic results, we can create a simple builder based on the nimbus-eth1 codebase that will be able to propose transactions from the regular network mempool. * Start the simulation from a merged state. This would allow us to start removing pre-merge functionality such as the gossip subsciption logic. The commit also removes the merge-forcing hack installed after the TTD removal. * Consolidate all the tools used in the local simulation into a single `ncli_testnet` binary.
2023-02-23 02:10:07 +00:00
extraData: DynamicBytes[0, MAX_EXTRA_DATA_BYTES](executionPayload.extra_data),
implement more missing capella functionality (#4344)
2022-11-24 07:53:04 +00:00
baseFeePerGas: executionPayload.base_fee_per_gas,
blockHash: executionPayload.block_hash.asBlockHash,
transactions: mapIt(executionPayload.transactions, it.getTypedTransaction))
Support for driving multiple EL nodes from a single Nimbus BN (#4465) * Support for driving multiple EL nodes from a single Nimbus BN Full list of changes: * Eth1Monitor has been renamed to ELManager to match its current responsibilities better. * The ELManager is no longer optional in the code (it won't have a nil value under any circumstances). * The support for subscribing for headers was removed as it only worked with WebSockets and contributed significant complexity while bringing only a very minor advantage. * The `--web3-url` parameter has been deprecated in favor of a new `--el` parameter. The new parameter has a reasonable default value and supports specifying a different JWT for each connection. Each connection can also be configured with a different set of responsibilities (e.g. download deposits, validate blocks and/or produce blocks). On the command-line, these properties can be configured through URL properties stored in the #anchor part of the URL. In TOML files, they come with a very natural syntax (althrough the URL scheme is also supported). * The previously scattered EL-related state and logic is now moved to `eth1_monitor.nim` (this module will be renamed to `el_manager.nim` in a follow-up commit). State is assigned properly either to the `ELManager` or the to individual `ELConnection` objects where appropriate. The ELManager executes all Engine API requests against all attached EL nodes, in parallel. It compares their results and if there is a disagreement regarding the validity of a certain payload, this is detected and the beacon node is protected from publishing a block with a potential execution layer consensus bug in it. The BN provides metrics per EL node for the number of successful or failed requests for each type Engine API requests. If an EL node goes offline and connectivity is resoted later, we report the problem and the remedy in edge-triggered fashion. * More progress towards implementing Deneb block production in the VC and comparing the value of blocks produced by the EL and the builder API. * Adds a Makefile target for the zhejiang testnet
2023-03-05 01:40:21 +00:00
template toEngineWithdrawal(w: capella.Withdrawal): WithdrawalV1 =
WithdrawalV1(
index: Quantity(w.index),
validatorIndex: Quantity(w.validator_index),
address: Address(w.address.data),
amount: Quantity(w.amount))
implement more missing capella functionality (#4344)
2022-11-24 07:53:04 +00:00
func asEngineExecutionPayload*(executionPayload: capella.ExecutionPayload):
support engine API execution payloads with withdrawals (#4358)
2022-11-29 05:02:16 +00:00
ExecutionPayloadV2 =
implement more missing capella functionality (#4344)
2022-11-24 07:53:04 +00:00
template getTypedTransaction(tt: bellatrix.Transaction): TypedTransaction =
TypedTransaction(tt.distinctBase)
support engine API execution payloads with withdrawals (#4358)
2022-11-29 05:02:16 +00:00
engine_api.ExecutionPayloadV2(
implement more missing capella functionality (#4344)
2022-11-24 07:53:04 +00:00
parentHash: executionPayload.parent_hash.asBlockHash,
feeRecipient: Address(executionPayload.fee_recipient.data),
stateRoot: executionPayload.state_root.asBlockHash,
receiptsRoot: executionPayload.receipts_root.asBlockHash,
logsBloom:
FixedBytes[BYTES_PER_LOGS_BLOOM](executionPayload.logs_bloom.data),
prevRandao: executionPayload.prev_randao.asBlockHash,
blockNumber: Quantity(executionPayload.block_number),
convert between engine and consensus ExecutionPayloads (#3228) * convert between engine and consensus ExecutionPayloads
2022-01-03 12:22:56 +00:00
gasLimit: Quantity(executionPayload.gas_limit),
gasUsed: Quantity(executionPayload.gas_used),
timestamp: Quantity(executionPayload.timestamp),
Local sim impovements (#4551) * Local sim impovements * Added support for running Capella and EIP-4844 simulations by downloading the correct version of Geth. * Added support for using Nimbus remote signer and Web3Signer. Use 2 out of 3 threshold signing configuration in the mainnet configuration and regular remote signing in the minimal one. * The local testnet simulation can now use a payload builder. This is currently not activated in CI due to lack of automated procedures for installing third-party relays or builders. You are adviced to use mergemock for now, but for most realistic results, we can create a simple builder based on the nimbus-eth1 codebase that will be able to propose transactions from the regular network mempool. * Start the simulation from a merged state. This would allow us to start removing pre-merge functionality such as the gossip subsciption logic. The commit also removes the merge-forcing hack installed after the TTD removal. * Consolidate all the tools used in the local simulation into a single `ncli_testnet` binary.
2023-02-23 02:10:07 +00:00
extraData: DynamicBytes[0, MAX_EXTRA_DATA_BYTES](executionPayload.extra_data),
fix ExecutionPayload(Header) JSON serialization (#3679)
2022-06-01 12:57:28 +00:00
baseFeePerGas: executionPayload.base_fee_per_gas,
convert between engine and consensus ExecutionPayloads (#3228) * convert between engine and consensus ExecutionPayloads
2022-01-03 12:22:56 +00:00
blockHash: executionPayload.block_hash.asBlockHash,
support engine API execution payloads with withdrawals (#4358)
2022-11-29 05:02:16 +00:00
transactions: mapIt(executionPayload.transactions, it.getTypedTransaction),
Support for driving multiple EL nodes from a single Nimbus BN (#4465) * Support for driving multiple EL nodes from a single Nimbus BN Full list of changes: * Eth1Monitor has been renamed to ELManager to match its current responsibilities better. * The ELManager is no longer optional in the code (it won't have a nil value under any circumstances). * The support for subscribing for headers was removed as it only worked with WebSockets and contributed significant complexity while bringing only a very minor advantage. * The `--web3-url` parameter has been deprecated in favor of a new `--el` parameter. The new parameter has a reasonable default value and supports specifying a different JWT for each connection. Each connection can also be configured with a different set of responsibilities (e.g. download deposits, validate blocks and/or produce blocks). On the command-line, these properties can be configured through URL properties stored in the #anchor part of the URL. In TOML files, they come with a very natural syntax (althrough the URL scheme is also supported). * The previously scattered EL-related state and logic is now moved to `eth1_monitor.nim` (this module will be renamed to `el_manager.nim` in a follow-up commit). State is assigned properly either to the `ELManager` or the to individual `ELConnection` objects where appropriate. The ELManager executes all Engine API requests against all attached EL nodes, in parallel. It compares their results and if there is a disagreement regarding the validity of a certain payload, this is detected and the beacon node is protected from publishing a block with a potential execution layer consensus bug in it. The BN provides metrics per EL node for the number of successful or failed requests for each type Engine API requests. If an EL node goes offline and connectivity is resoted later, we report the problem and the remedy in edge-triggered fashion. * More progress towards implementing Deneb block production in the VC and comparing the value of blocks produced by the EL and the builder API. * Adds a Makefile target for the zhejiang testnet
2023-03-05 01:40:21 +00:00
withdrawals: mapIt(executionPayload.withdrawals, it.toEngineWithdrawal))
convert between engine and consensus ExecutionPayloads (#3228) * convert between engine and consensus ExecutionPayloads
2022-01-03 12:22:56 +00:00
more eip4844 -> deneb changes (#4666)
2023-02-25 01:03:34 +00:00
func asEngineExecutionPayload*(executionPayload: deneb.ExecutionPayload):
Wire up engine_newPayloadV3 (#4482) * Wire up eip4844's newPayloadV3 * Add eip4844 test * Update AllTests-mainnet.md and fix typo
2023-01-11 18:21:19 +00:00
ExecutionPayloadV3 =
template getTypedTransaction(tt: bellatrix.Transaction): TypedTransaction =
TypedTransaction(tt.distinctBase)
engine_api.ExecutionPayloadV3(
parentHash: executionPayload.parent_hash.asBlockHash,
feeRecipient: Address(executionPayload.fee_recipient.data),
stateRoot: executionPayload.state_root.asBlockHash,
receiptsRoot: executionPayload.receipts_root.asBlockHash,
logsBloom:
FixedBytes[BYTES_PER_LOGS_BLOOM](executionPayload.logs_bloom.data),
prevRandao: executionPayload.prev_randao.asBlockHash,
blockNumber: Quantity(executionPayload.block_number),
gasLimit: Quantity(executionPayload.gas_limit),
gasUsed: Quantity(executionPayload.gas_used),
timestamp: Quantity(executionPayload.timestamp),
Local sim impovements (#4551) * Local sim impovements * Added support for running Capella and EIP-4844 simulations by downloading the correct version of Geth. * Added support for using Nimbus remote signer and Web3Signer. Use 2 out of 3 threshold signing configuration in the mainnet configuration and regular remote signing in the minimal one. * The local testnet simulation can now use a payload builder. This is currently not activated in CI due to lack of automated procedures for installing third-party relays or builders. You are adviced to use mergemock for now, but for most realistic results, we can create a simple builder based on the nimbus-eth1 codebase that will be able to propose transactions from the regular network mempool. * Start the simulation from a merged state. This would allow us to start removing pre-merge functionality such as the gossip subsciption logic. The commit also removes the merge-forcing hack installed after the TTD removal. * Consolidate all the tools used in the local simulation into a single `ncli_testnet` binary.
2023-02-23 02:10:07 +00:00
extraData: DynamicBytes[0, MAX_EXTRA_DATA_BYTES](executionPayload.extra_data),
Wire up engine_newPayloadV3 (#4482) * Wire up eip4844's newPayloadV3 * Add eip4844 test * Update AllTests-mainnet.md and fix typo
2023-01-11 18:21:19 +00:00
baseFeePerGas: executionPayload.base_fee_per_gas,
blockHash: executionPayload.block_hash.asBlockHash,
transactions: mapIt(executionPayload.transactions, it.getTypedTransaction),
`excess_data_gas` after `withdrawals` cleanup (#4900) We already updated the field order in the actual `ExecutionPayload`, but in init code and tests / logs etc we still used the old order. Update those occurrences to also match the field order in the struct. Furthermore, add `excess_data_gas` to last entry in `test_eth1_monitor`.
2023-05-05 23:15:47 +00:00
withdrawals: mapIt(executionPayload.withdrawals, it.asEngineWithdrawal),
Rename "data gas" to "blob gas" (#5216) * Rename data gas to blob gas * Update vendor/nim-eth and vendor/nim-web3
2023-08-02 22:07:57 +00:00
blobGasUsed: Quantity(executionPayload.blob_gas_used),
excessBlobGas: Quantity(executionPayload.excess_blob_gas))
Wire up engine_newPayloadV3 (#4482) * Wire up eip4844's newPayloadV3 * Add eip4844 test * Update AllTests-mainnet.md and fix typo
2023-01-11 18:21:19 +00:00
simplify libp2p logging (#1605) and a few other small logging fixes
2020-09-06 08:39:25 +00:00
func shortLog*(b: Eth1Block): string =
EH cleanup (#2455) almost 100% raises in nimbus-eth2 now! * fix some rare exception-related crashes in json-rpc
2021-03-26 06:52:01 +00:00
try:
Minor post-merge cleanups (#3945) https://github.com/status-im/nimbus-eth2/pull/3944 The use of nested `awaitWithRetries` calls would have resulted in an unexpected number of retries (3x3). We now use regular `await` in outer layer to avoid the problem. https://github.com/status-im/nimbus-eth2/pull/3943 The new code has an invariant that the `headMerkleizer` field in the `Eth1Chain` is always kept in sync with the blocks stored in the chain. This invariant is now enforced better by doing the necessary merkleizer updates in the `Eth1Chain.addBlock` function, in the `Eth1Chain.init` function and in the `Eth1Chain.reset` function.
2022-08-10 12:31:10 +00:00
&"{b.number}:{shortLog b.hash}(deposits = {b.depositCount})"
EH cleanup (#2455) almost 100% raises in nimbus-eth2 now! * fix some rare exception-related crashes in json-rpc
2021-03-26 06:52:01 +00:00
except ValueError as exc: raiseAssert exc.msg
Switch to monitoring of Blocks; More accurate genesis detection and faster Eth1 syncing
2020-06-27 12:01:19 +00:00
cleanups, partly from kintsugi branch (#3161) * cleanups, partly from kintsugi branch * re-export shortLog(EthBlock) and preserve exception messages in batchVerify and processBatch
2021-12-05 17:32:41 +00:00
template findBlock(chain: Eth1Chain, eth1Data: Eth1Data): Eth1Block =
Add Eth1 deposits simulation to block_sim
2020-12-03 04:30:35 +00:00
getOrDefault(chain.blocksByHash, asBlockHash(eth1Data.block_hash), nil)
Spec-compliant implementation of Eth1 monitoring; Eth1-enabled local sim BEWARE! This commit will trigger a stack overflow during local sim
2020-03-24 11:13:07 +00:00
Tentative fix for the Pyrmont Eth1 voting issue
2020-11-19 17:19:03 +00:00
func makeSuccessorWithoutDeposits(existingBlock: Eth1Block,
enable `styleCheck:usages` (#3573) Some upstream repos still need fixes, but this gets us close enough that style hints can be enabled by default. In general, "canonical" spellings are preferred even if they violate nep-1 - this applies in particular to spec-related stuff like `genesis_validators_root` which appears throughout the codebase.
2022-04-08 16:22:49 +00:00
successor: BlockObject): Eth1Block =
Fix the corrupted database state on Pyrmont nodes; Add mainnet genesis (#2056) * Handle some web3 timeouts better * Add support for developer .env files * Eth1 improvements; Mainnet genesis state Notable changes: * The deposits table have been removed from the database. The client will no longer process all deposits on start-up. * The network metadata now includes a "state snapshot" of the deposit contract. This allows the client to skip syncing deposits made prior to the snapshot (i.e. genesis). Suitable metadata added for Pyrmont and Mainnet. * The Eth1 monitor won't be started unless there are validators attached to the node. * The genesis detection code is now optional and disabled by default * Bugfix: The client should not produce blocks that will fail validation when it hasn't downloaded the latest deposits yet * Bugfix: Work around the database corruption affecting Pyrmont nodes * Remove metadata for Toledo and Medalla
2020-11-24 21:21:47 +00:00
result = Eth1Block(
Minor post-merge cleanups (#3945) https://github.com/status-im/nimbus-eth2/pull/3944 The use of nested `awaitWithRetries` calls would have resulted in an unexpected number of retries (3x3). We now use regular `await` in outer layer to avoid the problem. https://github.com/status-im/nimbus-eth2/pull/3943 The new code has an invariant that the `headMerkleizer` field in the `Eth1Chain` is always kept in sync with the blocks stored in the chain. This invariant is now enforced better by doing the necessary merkleizer updates in the `Eth1Chain.addBlock` function, in the `Eth1Chain.init` function and in the `Eth1Chain.reset` function.
2022-08-10 12:31:10 +00:00
hash: successor.hash.asEth2Digest,
Tentative fix for the Pyrmont Eth1 voting issue
2020-11-19 17:19:03 +00:00
number: Eth1BlockNumber successor.number,
Minor post-merge cleanups (#3945) https://github.com/status-im/nimbus-eth2/pull/3944 The use of nested `awaitWithRetries` calls would have resulted in an unexpected number of retries (3x3). We now use regular `await` in outer layer to avoid the problem. https://github.com/status-im/nimbus-eth2/pull/3943 The new code has an invariant that the `headMerkleizer` field in the `Eth1Chain` is always kept in sync with the blocks stored in the chain. This invariant is now enforced better by doing the necessary merkleizer updates in the `Eth1Chain.addBlock` function, in the `Eth1Chain.init` function and in the `Eth1Chain.reset` function.
2022-08-10 12:31:10 +00:00
timestamp: Eth1BlockTimestamp successor.timestamp)
Fix the corrupted database state on Pyrmont nodes; Add mainnet genesis (#2056) * Handle some web3 timeouts better * Add support for developer .env files * Eth1 improvements; Mainnet genesis state Notable changes: * The deposits table have been removed from the database. The client will no longer process all deposits on start-up. * The network metadata now includes a "state snapshot" of the deposit contract. This allows the client to skip syncing deposits made prior to the snapshot (i.e. genesis). Suitable metadata added for Pyrmont and Mainnet. * The Eth1 monitor won't be started unless there are validators attached to the node. * The genesis detection code is now optional and disabled by default * Bugfix: The client should not produce blocks that will fail validation when it hasn't downloaded the latest deposits yet * Bugfix: Work around the database corruption affecting Pyrmont nodes * Remove metadata for Toledo and Medalla
2020-11-24 21:21:47 +00:00
Add Eth1 deposits simulation to block_sim
2020-12-03 04:30:35 +00:00
func latestCandidateBlock(chain: Eth1Chain, periodStart: uint64): Eth1Block =
for i in countdown(chain.blocks.len - 1, 0):
let blk = chain.blocks[i]
Implement split preset/config support (#2710) * Implement split preset/config support This is the initial bulk refactor to introduce runtime config values in a number of places, somewhat replacing the existing mechanism of loading network metadata. It still needs more work, this is the initial refactor that introduces runtime configuration in some of the places that need it. The PR changes the way presets and constants work, to match the spec. In particular, a "preset" now refers to the compile-time configuration while a "cfg" or "RuntimeConfig" is the dynamic part. A single binary can support either mainnet or minimal, but not both. Support for other presets has been removed completely (can be readded, in case there's need). There's a number of outstanding tasks: * `SECONDS_PER_SLOT` still needs fixing * loading custom runtime configs needs redoing * checking constants against YAML file * yeerongpilly support `build/nimbus_beacon_node --network=yeerongpilly --discv5:no --log-level=DEBUG` * load fork epoch from config * fix fork digest sent in status * nicer error string for request failures * fix tools * one more * fixup * fixup * fixup * use "standard" network definition folder in local testnet Files are loaded from their standard locations, including genesis etc, to conform to the format used in the `eth2-networks` repo. * fix launch scripts, allow unknown config values * fix base config of rest test * cleanups * bundle mainnet config using common loader * fix spec links and names * only include supported preset in binary * drop yeerongpilly, add altair-devnet-0, support boot_enr.yaml
2021-07-12 13:01:38 +00:00
if is_candidate_block(chain.cfg, blk, periodStart):
Spec-compliant implementation of Eth1 monitoring; Eth1-enabled local sim BEWARE! This commit will trigger a stack overflow during local sim
2020-03-24 11:13:07 +00:00
return blk
Add Eth1 deposits simulation to block_sim
2020-12-03 04:30:35 +00:00
proc popFirst(chain: var Eth1Chain) =
let removed = chain.blocks.popFirst
Minor post-merge cleanups (#3945) https://github.com/status-im/nimbus-eth2/pull/3944 The use of nested `awaitWithRetries` calls would have resulted in an unexpected number of retries (3x3). We now use regular `await` in outer layer to avoid the problem. https://github.com/status-im/nimbus-eth2/pull/3943 The new code has an invariant that the `headMerkleizer` field in the `Eth1Chain` is always kept in sync with the blocks stored in the chain. This invariant is now enforced better by doing the necessary merkleizer updates in the `Eth1Chain.addBlock` function, in the `Eth1Chain.init` function and in the `Eth1Chain.reset` function.
2022-08-10 12:31:10 +00:00
chain.blocksByHash.del removed.hash.asBlockHash
Add Eth1 deposits simulation to block_sim
2020-12-03 04:30:35 +00:00
eth1_chain_len.set chain.blocks.len.int64
Switch to monitoring of Blocks; More accurate genesis detection and faster Eth1 syncing
2020-06-27 12:01:19 +00:00
ssz: bump (#5463) more `zeroMem` avoidance, value semantics for merkleizer
2023-09-28 11:59:33 +00:00
func getDepositsRoot*(m: var DepositsMerkleizer): Eth2Digest =
Minor post-merge cleanups (#3945) https://github.com/status-im/nimbus-eth2/pull/3944 The use of nested `awaitWithRetries` calls would have resulted in an unexpected number of retries (3x3). We now use regular `await` in outer layer to avoid the problem. https://github.com/status-im/nimbus-eth2/pull/3943 The new code has an invariant that the `headMerkleizer` field in the `Eth1Chain` is always kept in sync with the blocks stored in the chain. This invariant is now enforced better by doing the necessary merkleizer updates in the `Eth1Chain.addBlock` function, in the `Eth1Chain.init` function and in the `Eth1Chain.reset` function.
2022-08-10 12:31:10 +00:00
mixInLength(m.getFinalHash, int m.totalChunks)
Add Eth1 deposits simulation to block_sim
2020-12-03 04:30:35 +00:00
proc addBlock*(chain: var Eth1Chain, newBlock: Eth1Block) =
Minor post-merge cleanups (#3945) https://github.com/status-im/nimbus-eth2/pull/3944 The use of nested `awaitWithRetries` calls would have resulted in an unexpected number of retries (3x3). We now use regular `await` in outer layer to avoid the problem. https://github.com/status-im/nimbus-eth2/pull/3943 The new code has an invariant that the `headMerkleizer` field in the `Eth1Chain` is always kept in sync with the blocks stored in the chain. This invariant is now enforced better by doing the necessary merkleizer updates in the `Eth1Chain.addBlock` function, in the `Eth1Chain.init` function and in the `Eth1Chain.reset` function.
2022-08-10 12:31:10 +00:00
for deposit in newBlock.deposits:
chain.headMerkleizer.addChunk hash_tree_root(deposit).data
newBlock.depositCount = chain.headMerkleizer.getChunkCount
newBlock.depositRoot = chain.headMerkleizer.getDepositsRoot
Add Eth1 deposits simulation to block_sim
2020-12-03 04:30:35 +00:00
chain.blocks.addLast newBlock
Minor post-merge cleanups (#3945) https://github.com/status-im/nimbus-eth2/pull/3944 The use of nested `awaitWithRetries` calls would have resulted in an unexpected number of retries (3x3). We now use regular `await` in outer layer to avoid the problem. https://github.com/status-im/nimbus-eth2/pull/3943 The new code has an invariant that the `headMerkleizer` field in the `Eth1Chain` is always kept in sync with the blocks stored in the chain. This invariant is now enforced better by doing the necessary merkleizer updates in the `Eth1Chain.addBlock` function, in the `Eth1Chain.init` function and in the `Eth1Chain.reset` function.
2022-08-10 12:31:10 +00:00
chain.blocksByHash[newBlock.hash.asBlockHash] = newBlock
Add Eth1 deposits simulation to block_sim
2020-12-03 04:30:35 +00:00
eth1_chain_len.set chain.blocks.len.int64
Spec-compliant implementation of Eth1 monitoring; Eth1-enabled local sim BEWARE! This commit will trigger a stack overflow during local sim
2020-03-24 11:13:07 +00:00
Minor post-merge cleanups (#3945) https://github.com/status-im/nimbus-eth2/pull/3944 The use of nested `awaitWithRetries` calls would have resulted in an unexpected number of retries (3x3). We now use regular `await` in outer layer to avoid the problem. https://github.com/status-im/nimbus-eth2/pull/3943 The new code has an invariant that the `headMerkleizer` field in the `Eth1Chain` is always kept in sync with the blocks stored in the chain. This invariant is now enforced better by doing the necessary merkleizer updates in the `Eth1Chain.addBlock` function, in the `Eth1Chain.init` function and in the `Eth1Chain.reset` function.
2022-08-10 12:31:10 +00:00
func toVoteData(blk: Eth1Block): Eth1Data =
Eth1Data(
deposit_root: blk.depositRoot,
deposit_count: blk.depositCount,
block_hash: blk.hash)
Merge BeaconBlock gossip validation (#3165) * Merge BeaconBlock gossip validation * figure/ground inversion * revert cosmetic cleanups to reduce merge conflicts
2021-12-08 17:29:22 +00:00
func hash*(x: Eth1Data): Hash =
More efficient implementation of the 'POST beacon_committee_subscriptions' API (#3153)
2021-12-03 15:04:58 +00:00
hash(x.block_hash)
Tentative fix for the Pyrmont Eth1 voting issue
2020-11-19 17:19:03 +00:00
simplify ELMonitor fcU payload attributes handling (#4696)
2023-03-06 16:19:15 +00:00
func isConnected(connection: ELConnection): bool =
Support for driving multiple EL nodes from a single Nimbus BN (#4465) * Support for driving multiple EL nodes from a single Nimbus BN Full list of changes: * Eth1Monitor has been renamed to ELManager to match its current responsibilities better. * The ELManager is no longer optional in the code (it won't have a nil value under any circumstances). * The support for subscribing for headers was removed as it only worked with WebSockets and contributed significant complexity while bringing only a very minor advantage. * The `--web3-url` parameter has been deprecated in favor of a new `--el` parameter. The new parameter has a reasonable default value and supports specifying a different JWT for each connection. Each connection can also be configured with a different set of responsibilities (e.g. download deposits, validate blocks and/or produce blocks). On the command-line, these properties can be configured through URL properties stored in the #anchor part of the URL. In TOML files, they come with a very natural syntax (althrough the URL scheme is also supported). * The previously scattered EL-related state and logic is now moved to `eth1_monitor.nim` (this module will be renamed to `el_manager.nim` in a follow-up commit). State is assigned properly either to the `ELManager` or the to individual `ELConnection` objects where appropriate. The ELManager executes all Engine API requests against all attached EL nodes, in parallel. It compares their results and if there is a disagreement regarding the validity of a certain payload, this is detected and the beacon node is protected from publishing a block with a potential execution layer consensus bug in it. The BN provides metrics per EL node for the number of successful or failed requests for each type Engine API requests. If an EL node goes offline and connectivity is resoted later, we report the problem and the remedy in edge-triggered fashion. * More progress towards implementing Deneb block production in the VC and comparing the value of blocks produced by the EL and the builder API. * Adds a Makefile target for the zhejiang testnet
2023-03-05 01:40:21 +00:00
connection.web3.isSome
simplify ELMonitor fcU payload attributes handling (#4696)
2023-03-06 16:19:15 +00:00
func getJsonRpcRequestHeaders(jwtSecret: Option[seq[byte]]):
Support for driving multiple EL nodes from a single Nimbus BN (#4465) * Support for driving multiple EL nodes from a single Nimbus BN Full list of changes: * Eth1Monitor has been renamed to ELManager to match its current responsibilities better. * The ELManager is no longer optional in the code (it won't have a nil value under any circumstances). * The support for subscribing for headers was removed as it only worked with WebSockets and contributed significant complexity while bringing only a very minor advantage. * The `--web3-url` parameter has been deprecated in favor of a new `--el` parameter. The new parameter has a reasonable default value and supports specifying a different JWT for each connection. Each connection can also be configured with a different set of responsibilities (e.g. download deposits, validate blocks and/or produce blocks). On the command-line, these properties can be configured through URL properties stored in the #anchor part of the URL. In TOML files, they come with a very natural syntax (althrough the URL scheme is also supported). * The previously scattered EL-related state and logic is now moved to `eth1_monitor.nim` (this module will be renamed to `el_manager.nim` in a follow-up commit). State is assigned properly either to the `ELManager` or the to individual `ELConnection` objects where appropriate. The ELManager executes all Engine API requests against all attached EL nodes, in parallel. It compares their results and if there is a disagreement regarding the validity of a certain payload, this is detected and the beacon node is protected from publishing a block with a potential execution layer consensus bug in it. The BN provides metrics per EL node for the number of successful or failed requests for each type Engine API requests. If an EL node goes offline and connectivity is resoted later, we report the problem and the remedy in edge-triggered fashion. * More progress towards implementing Deneb block production in the VC and comparing the value of blocks produced by the EL and the builder API. * Adds a Makefile target for the zhejiang testnet
2023-03-05 01:40:21 +00:00
auto =
if jwtSecret.isSome:
let secret = jwtSecret.get
(proc(): seq[(string, string)] =
# https://www.rfc-editor.org/rfc/rfc6750#section-6.1.1
@[("Authorization", "Bearer " & getSignedIatToken(
secret, (getTime() - initTime(0, 0)).inSeconds))])
else:
(proc(): seq[(string, string)] = @[])
proc newWeb3*(engineUrl: EngineApiUrl): Future[Web3] =
Always use fresh HTTP connections for EL communication
2023-05-04 23:27:26 +00:00
newWeb3(engineUrl.url,
getJsonRpcRequestHeaders(engineUrl.jwtSecret),
httpFlags = {HttpClientFlag.NewConnectionAlways})
Support for driving multiple EL nodes from a single Nimbus BN (#4465) * Support for driving multiple EL nodes from a single Nimbus BN Full list of changes: * Eth1Monitor has been renamed to ELManager to match its current responsibilities better. * The ELManager is no longer optional in the code (it won't have a nil value under any circumstances). * The support for subscribing for headers was removed as it only worked with WebSockets and contributed significant complexity while bringing only a very minor advantage. * The `--web3-url` parameter has been deprecated in favor of a new `--el` parameter. The new parameter has a reasonable default value and supports specifying a different JWT for each connection. Each connection can also be configured with a different set of responsibilities (e.g. download deposits, validate blocks and/or produce blocks). On the command-line, these properties can be configured through URL properties stored in the #anchor part of the URL. In TOML files, they come with a very natural syntax (althrough the URL scheme is also supported). * The previously scattered EL-related state and logic is now moved to `eth1_monitor.nim` (this module will be renamed to `el_manager.nim` in a follow-up commit). State is assigned properly either to the `ELManager` or the to individual `ELConnection` objects where appropriate. The ELManager executes all Engine API requests against all attached EL nodes, in parallel. It compares their results and if there is a disagreement regarding the validity of a certain payload, this is detected and the beacon node is protected from publishing a block with a potential execution layer consensus bug in it. The BN provides metrics per EL node for the number of successful or failed requests for each type Engine API requests. If an EL node goes offline and connectivity is resoted later, we report the problem and the remedy in edge-triggered fashion. * More progress towards implementing Deneb block production in the VC and comparing the value of blocks produced by the EL and the builder API. * Adds a Makefile target for the zhejiang testnet
2023-03-05 01:40:21 +00:00
proc establishEngineApiConnection*(url: EngineApiUrl):
Future[Result[Web3, string]] {.async.} =
let web3Fut = newWeb3(url)
yield web3Fut or sleepAsync(engineApiConnectionTimeout)
More forceful EL connection resetting (fixes #4424) (#4708)
2023-03-09 23:41:28 +00:00
if not web3Fut.completed:
Support for driving multiple EL nodes from a single Nimbus BN (#4465) * Support for driving multiple EL nodes from a single Nimbus BN Full list of changes: * Eth1Monitor has been renamed to ELManager to match its current responsibilities better. * The ELManager is no longer optional in the code (it won't have a nil value under any circumstances). * The support for subscribing for headers was removed as it only worked with WebSockets and contributed significant complexity while bringing only a very minor advantage. * The `--web3-url` parameter has been deprecated in favor of a new `--el` parameter. The new parameter has a reasonable default value and supports specifying a different JWT for each connection. Each connection can also be configured with a different set of responsibilities (e.g. download deposits, validate blocks and/or produce blocks). On the command-line, these properties can be configured through URL properties stored in the #anchor part of the URL. In TOML files, they come with a very natural syntax (althrough the URL scheme is also supported). * The previously scattered EL-related state and logic is now moved to `eth1_monitor.nim` (this module will be renamed to `el_manager.nim` in a follow-up commit). State is assigned properly either to the `ELManager` or the to individual `ELConnection` objects where appropriate. The ELManager executes all Engine API requests against all attached EL nodes, in parallel. It compares their results and if there is a disagreement regarding the validity of a certain payload, this is detected and the beacon node is protected from publishing a block with a potential execution layer consensus bug in it. The BN provides metrics per EL node for the number of successful or failed requests for each type Engine API requests. If an EL node goes offline and connectivity is resoted later, we report the problem and the remedy in edge-triggered fashion. * More progress towards implementing Deneb block production in the VC and comparing the value of blocks produced by the EL and the builder API. * Adds a Makefile target for the zhejiang testnet
2023-03-05 01:40:21 +00:00
await cancelAndWait(web3Fut)
if web3Fut.failed:
return err "Failed to setup Engine API connection: " & web3Fut.readError.msg
Retry all web3 requests up to 3 times with exponential backoff
2020-12-01 21:20:28 +00:00
else:
Support for driving multiple EL nodes from a single Nimbus BN (#4465) * Support for driving multiple EL nodes from a single Nimbus BN Full list of changes: * Eth1Monitor has been renamed to ELManager to match its current responsibilities better. * The ELManager is no longer optional in the code (it won't have a nil value under any circumstances). * The support for subscribing for headers was removed as it only worked with WebSockets and contributed significant complexity while bringing only a very minor advantage. * The `--web3-url` parameter has been deprecated in favor of a new `--el` parameter. The new parameter has a reasonable default value and supports specifying a different JWT for each connection. Each connection can also be configured with a different set of responsibilities (e.g. download deposits, validate blocks and/or produce blocks). On the command-line, these properties can be configured through URL properties stored in the #anchor part of the URL. In TOML files, they come with a very natural syntax (althrough the URL scheme is also supported). * The previously scattered EL-related state and logic is now moved to `eth1_monitor.nim` (this module will be renamed to `el_manager.nim` in a follow-up commit). State is assigned properly either to the `ELManager` or the to individual `ELConnection` objects where appropriate. The ELManager executes all Engine API requests against all attached EL nodes, in parallel. It compares their results and if there is a disagreement regarding the validity of a certain payload, this is detected and the beacon node is protected from publishing a block with a potential execution layer consensus bug in it. The BN provides metrics per EL node for the number of successful or failed requests for each type Engine API requests. If an EL node goes offline and connectivity is resoted later, we report the problem and the remedy in edge-triggered fashion. * More progress towards implementing Deneb block production in the VC and comparing the value of blocks produced by the EL and the builder API. * Adds a Makefile target for the zhejiang testnet
2023-03-05 01:40:21 +00:00
return err "Failed to setup Engine API connection"
else:
return ok web3Fut.read
Retry all web3 requests up to 3 times with exponential backoff
2020-12-01 21:20:28 +00:00
Support for driving multiple EL nodes from a single Nimbus BN (#4465) * Support for driving multiple EL nodes from a single Nimbus BN Full list of changes: * Eth1Monitor has been renamed to ELManager to match its current responsibilities better. * The ELManager is no longer optional in the code (it won't have a nil value under any circumstances). * The support for subscribing for headers was removed as it only worked with WebSockets and contributed significant complexity while bringing only a very minor advantage. * The `--web3-url` parameter has been deprecated in favor of a new `--el` parameter. The new parameter has a reasonable default value and supports specifying a different JWT for each connection. Each connection can also be configured with a different set of responsibilities (e.g. download deposits, validate blocks and/or produce blocks). On the command-line, these properties can be configured through URL properties stored in the #anchor part of the URL. In TOML files, they come with a very natural syntax (althrough the URL scheme is also supported). * The previously scattered EL-related state and logic is now moved to `eth1_monitor.nim` (this module will be renamed to `el_manager.nim` in a follow-up commit). State is assigned properly either to the `ELManager` or the to individual `ELConnection` objects where appropriate. The ELManager executes all Engine API requests against all attached EL nodes, in parallel. It compares their results and if there is a disagreement regarding the validity of a certain payload, this is detected and the beacon node is protected from publishing a block with a potential execution layer consensus bug in it. The BN provides metrics per EL node for the number of successful or failed requests for each type Engine API requests. If an EL node goes offline and connectivity is resoted later, we report the problem and the remedy in edge-triggered fashion. * More progress towards implementing Deneb block production in the VC and comparing the value of blocks produced by the EL and the builder API. * Adds a Makefile target for the zhejiang testnet
2023-03-05 01:40:21 +00:00
proc tryConnecting(connection: ELConnection): Future[bool] {.async.} =
if connection.isConnected:
return true
Retry all web3 requests up to 3 times with exponential backoff
2020-12-01 21:20:28 +00:00
More forceful EL connection resetting (fixes #4424) (#4708)
2023-03-09 23:41:28 +00:00
if connection.connectingFut == nil or
connection.connectingFut.finished: # The previous attempt was not successful
Support for driving multiple EL nodes from a single Nimbus BN (#4465) * Support for driving multiple EL nodes from a single Nimbus BN Full list of changes: * Eth1Monitor has been renamed to ELManager to match its current responsibilities better. * The ELManager is no longer optional in the code (it won't have a nil value under any circumstances). * The support for subscribing for headers was removed as it only worked with WebSockets and contributed significant complexity while bringing only a very minor advantage. * The `--web3-url` parameter has been deprecated in favor of a new `--el` parameter. The new parameter has a reasonable default value and supports specifying a different JWT for each connection. Each connection can also be configured with a different set of responsibilities (e.g. download deposits, validate blocks and/or produce blocks). On the command-line, these properties can be configured through URL properties stored in the #anchor part of the URL. In TOML files, they come with a very natural syntax (althrough the URL scheme is also supported). * The previously scattered EL-related state and logic is now moved to `eth1_monitor.nim` (this module will be renamed to `el_manager.nim` in a follow-up commit). State is assigned properly either to the `ELManager` or the to individual `ELConnection` objects where appropriate. The ELManager executes all Engine API requests against all attached EL nodes, in parallel. It compares their results and if there is a disagreement regarding the validity of a certain payload, this is detected and the beacon node is protected from publishing a block with a potential execution layer consensus bug in it. The BN provides metrics per EL node for the number of successful or failed requests for each type Engine API requests. If an EL node goes offline and connectivity is resoted later, we report the problem and the remedy in edge-triggered fashion. * More progress towards implementing Deneb block production in the VC and comparing the value of blocks produced by the EL and the builder API. * Adds a Makefile target for the zhejiang testnet
2023-03-05 01:40:21 +00:00
connection.connectingFut = establishEngineApiConnection(connection.engineUrl)
Retry all web3 requests up to 3 times with exponential backoff
2020-12-01 21:20:28 +00:00
Support for driving multiple EL nodes from a single Nimbus BN (#4465) * Support for driving multiple EL nodes from a single Nimbus BN Full list of changes: * Eth1Monitor has been renamed to ELManager to match its current responsibilities better. * The ELManager is no longer optional in the code (it won't have a nil value under any circumstances). * The support for subscribing for headers was removed as it only worked with WebSockets and contributed significant complexity while bringing only a very minor advantage. * The `--web3-url` parameter has been deprecated in favor of a new `--el` parameter. The new parameter has a reasonable default value and supports specifying a different JWT for each connection. Each connection can also be configured with a different set of responsibilities (e.g. download deposits, validate blocks and/or produce blocks). On the command-line, these properties can be configured through URL properties stored in the #anchor part of the URL. In TOML files, they come with a very natural syntax (althrough the URL scheme is also supported). * The previously scattered EL-related state and logic is now moved to `eth1_monitor.nim` (this module will be renamed to `el_manager.nim` in a follow-up commit). State is assigned properly either to the `ELManager` or the to individual `ELConnection` objects where appropriate. The ELManager executes all Engine API requests against all attached EL nodes, in parallel. It compares their results and if there is a disagreement regarding the validity of a certain payload, this is detected and the beacon node is protected from publishing a block with a potential execution layer consensus bug in it. The BN provides metrics per EL node for the number of successful or failed requests for each type Engine API requests. If an EL node goes offline and connectivity is resoted later, we report the problem and the remedy in edge-triggered fashion. * More progress towards implementing Deneb block production in the VC and comparing the value of blocks produced by the EL and the builder API. * Adds a Makefile target for the zhejiang testnet
2023-03-05 01:40:21 +00:00
let web3Res = await connection.connectingFut
if web3Res.isErr:
return false
else:
connection.web3 = some web3Res.get
return true
Retry all web3 requests up to 3 times with exponential backoff
2020-12-01 21:20:28 +00:00
Support for driving multiple EL nodes from a single Nimbus BN (#4465) * Support for driving multiple EL nodes from a single Nimbus BN Full list of changes: * Eth1Monitor has been renamed to ELManager to match its current responsibilities better. * The ELManager is no longer optional in the code (it won't have a nil value under any circumstances). * The support for subscribing for headers was removed as it only worked with WebSockets and contributed significant complexity while bringing only a very minor advantage. * The `--web3-url` parameter has been deprecated in favor of a new `--el` parameter. The new parameter has a reasonable default value and supports specifying a different JWT for each connection. Each connection can also be configured with a different set of responsibilities (e.g. download deposits, validate blocks and/or produce blocks). On the command-line, these properties can be configured through URL properties stored in the #anchor part of the URL. In TOML files, they come with a very natural syntax (althrough the URL scheme is also supported). * The previously scattered EL-related state and logic is now moved to `eth1_monitor.nim` (this module will be renamed to `el_manager.nim` in a follow-up commit). State is assigned properly either to the `ELManager` or the to individual `ELConnection` objects where appropriate. The ELManager executes all Engine API requests against all attached EL nodes, in parallel. It compares their results and if there is a disagreement regarding the validity of a certain payload, this is detected and the beacon node is protected from publishing a block with a potential execution layer consensus bug in it. The BN provides metrics per EL node for the number of successful or failed requests for each type Engine API requests. If an EL node goes offline and connectivity is resoted later, we report the problem and the remedy in edge-triggered fashion. * More progress towards implementing Deneb block production in the VC and comparing the value of blocks produced by the EL and the builder API. * Adds a Makefile target for the zhejiang testnet
2023-03-05 01:40:21 +00:00
proc connectedRpcClient(connection: ELConnection): Future[RpcClient] {.async.} =
while not connection.isConnected:
if not await connection.tryConnecting():
await sleepAsync(chronos.seconds(10))
Switch to monitoring of Blocks; More accurate genesis detection and faster Eth1 syncing
2020-06-27 12:01:19 +00:00
Support for driving multiple EL nodes from a single Nimbus BN (#4465) * Support for driving multiple EL nodes from a single Nimbus BN Full list of changes: * Eth1Monitor has been renamed to ELManager to match its current responsibilities better. * The ELManager is no longer optional in the code (it won't have a nil value under any circumstances). * The support for subscribing for headers was removed as it only worked with WebSockets and contributed significant complexity while bringing only a very minor advantage. * The `--web3-url` parameter has been deprecated in favor of a new `--el` parameter. The new parameter has a reasonable default value and supports specifying a different JWT for each connection. Each connection can also be configured with a different set of responsibilities (e.g. download deposits, validate blocks and/or produce blocks). On the command-line, these properties can be configured through URL properties stored in the #anchor part of the URL. In TOML files, they come with a very natural syntax (althrough the URL scheme is also supported). * The previously scattered EL-related state and logic is now moved to `eth1_monitor.nim` (this module will be renamed to `el_manager.nim` in a follow-up commit). State is assigned properly either to the `ELManager` or the to individual `ELConnection` objects where appropriate. The ELManager executes all Engine API requests against all attached EL nodes, in parallel. It compares their results and if there is a disagreement regarding the validity of a certain payload, this is detected and the beacon node is protected from publishing a block with a potential execution layer consensus bug in it. The BN provides metrics per EL node for the number of successful or failed requests for each type Engine API requests. If an EL node goes offline and connectivity is resoted later, we report the problem and the remedy in edge-triggered fashion. * More progress towards implementing Deneb block production in the VC and comparing the value of blocks produced by the EL and the builder API. * Adds a Makefile target for the zhejiang testnet
2023-03-05 01:40:21 +00:00
return connection.web3.get.provider
Switch to monitoring of Blocks; More accurate genesis detection and faster Eth1 syncing
2020-06-27 12:01:19 +00:00
Support for driving multiple EL nodes from a single Nimbus BN (#4465) * Support for driving multiple EL nodes from a single Nimbus BN Full list of changes: * Eth1Monitor has been renamed to ELManager to match its current responsibilities better. * The ELManager is no longer optional in the code (it won't have a nil value under any circumstances). * The support for subscribing for headers was removed as it only worked with WebSockets and contributed significant complexity while bringing only a very minor advantage. * The `--web3-url` parameter has been deprecated in favor of a new `--el` parameter. The new parameter has a reasonable default value and supports specifying a different JWT for each connection. Each connection can also be configured with a different set of responsibilities (e.g. download deposits, validate blocks and/or produce blocks). On the command-line, these properties can be configured through URL properties stored in the #anchor part of the URL. In TOML files, they come with a very natural syntax (althrough the URL scheme is also supported). * The previously scattered EL-related state and logic is now moved to `eth1_monitor.nim` (this module will be renamed to `el_manager.nim` in a follow-up commit). State is assigned properly either to the `ELManager` or the to individual `ELConnection` objects where appropriate. The ELManager executes all Engine API requests against all attached EL nodes, in parallel. It compares their results and if there is a disagreement regarding the validity of a certain payload, this is detected and the beacon node is protected from publishing a block with a potential execution layer consensus bug in it. The BN provides metrics per EL node for the number of successful or failed requests for each type Engine API requests. If an EL node goes offline and connectivity is resoted later, we report the problem and the remedy in edge-triggered fashion. * More progress towards implementing Deneb block production in the VC and comparing the value of blocks produced by the EL and the builder API. * Adds a Makefile target for the zhejiang testnet
2023-03-05 01:40:21 +00:00
proc getBlockByHash(rpcClient: RpcClient, hash: BlockHash): Future[BlockObject] =
rpcClient.eth_getBlockByHash(hash, false)
Switch to monitoring of Blocks; More accurate genesis detection and faster Eth1 syncing
2020-06-27 12:01:19 +00:00
Support for driving multiple EL nodes from a single Nimbus BN (#4465) * Support for driving multiple EL nodes from a single Nimbus BN Full list of changes: * Eth1Monitor has been renamed to ELManager to match its current responsibilities better. * The ELManager is no longer optional in the code (it won't have a nil value under any circumstances). * The support for subscribing for headers was removed as it only worked with WebSockets and contributed significant complexity while bringing only a very minor advantage. * The `--web3-url` parameter has been deprecated in favor of a new `--el` parameter. The new parameter has a reasonable default value and supports specifying a different JWT for each connection. Each connection can also be configured with a different set of responsibilities (e.g. download deposits, validate blocks and/or produce blocks). On the command-line, these properties can be configured through URL properties stored in the #anchor part of the URL. In TOML files, they come with a very natural syntax (althrough the URL scheme is also supported). * The previously scattered EL-related state and logic is now moved to `eth1_monitor.nim` (this module will be renamed to `el_manager.nim` in a follow-up commit). State is assigned properly either to the `ELManager` or the to individual `ELConnection` objects where appropriate. The ELManager executes all Engine API requests against all attached EL nodes, in parallel. It compares their results and if there is a disagreement regarding the validity of a certain payload, this is detected and the beacon node is protected from publishing a block with a potential execution layer consensus bug in it. The BN provides metrics per EL node for the number of successful or failed requests for each type Engine API requests. If an EL node goes offline and connectivity is resoted later, we report the problem and the remedy in edge-triggered fashion. * More progress towards implementing Deneb block production in the VC and comparing the value of blocks produced by the EL and the builder API. * Adds a Makefile target for the zhejiang testnet
2023-03-05 01:40:21 +00:00
proc getBlockByNumber*(rpcClient: RpcClient,
update to engine API alpha.6 (#3351)
2022-02-04 12:12:19 +00:00
number: Eth1BlockNumber): Future[BlockObject] =
Support for driving multiple EL nodes from a single Nimbus BN (#4465) * Support for driving multiple EL nodes from a single Nimbus BN Full list of changes: * Eth1Monitor has been renamed to ELManager to match its current responsibilities better. * The ELManager is no longer optional in the code (it won't have a nil value under any circumstances). * The support for subscribing for headers was removed as it only worked with WebSockets and contributed significant complexity while bringing only a very minor advantage. * The `--web3-url` parameter has been deprecated in favor of a new `--el` parameter. The new parameter has a reasonable default value and supports specifying a different JWT for each connection. Each connection can also be configured with a different set of responsibilities (e.g. download deposits, validate blocks and/or produce blocks). On the command-line, these properties can be configured through URL properties stored in the #anchor part of the URL. In TOML files, they come with a very natural syntax (althrough the URL scheme is also supported). * The previously scattered EL-related state and logic is now moved to `eth1_monitor.nim` (this module will be renamed to `el_manager.nim` in a follow-up commit). State is assigned properly either to the `ELManager` or the to individual `ELConnection` objects where appropriate. The ELManager executes all Engine API requests against all attached EL nodes, in parallel. It compares their results and if there is a disagreement regarding the validity of a certain payload, this is detected and the beacon node is protected from publishing a block with a potential execution layer consensus bug in it. The BN provides metrics per EL node for the number of successful or failed requests for each type Engine API requests. If an EL node goes offline and connectivity is resoted later, we report the problem and the remedy in edge-triggered fashion. * More progress towards implementing Deneb block production in the VC and comparing the value of blocks produced by the EL and the builder API. * Adds a Makefile target for the zhejiang testnet
2023-03-05 01:40:21 +00:00
let hexNumber = try:
&"0x{number:X}" # No leading 0's!
except ValueError as exc:
# Since the format above is valid, failing here should not be possible
raiseAssert exc.msg
Add stubs for EIP4844 engine API calls (#4536)
2023-01-21 00:47:38 +00:00
Support for driving multiple EL nodes from a single Nimbus BN (#4465) * Support for driving multiple EL nodes from a single Nimbus BN Full list of changes: * Eth1Monitor has been renamed to ELManager to match its current responsibilities better. * The ELManager is no longer optional in the code (it won't have a nil value under any circumstances). * The support for subscribing for headers was removed as it only worked with WebSockets and contributed significant complexity while bringing only a very minor advantage. * The `--web3-url` parameter has been deprecated in favor of a new `--el` parameter. The new parameter has a reasonable default value and supports specifying a different JWT for each connection. Each connection can also be configured with a different set of responsibilities (e.g. download deposits, validate blocks and/or produce blocks). On the command-line, these properties can be configured through URL properties stored in the #anchor part of the URL. In TOML files, they come with a very natural syntax (althrough the URL scheme is also supported). * The previously scattered EL-related state and logic is now moved to `eth1_monitor.nim` (this module will be renamed to `el_manager.nim` in a follow-up commit). State is assigned properly either to the `ELManager` or the to individual `ELConnection` objects where appropriate. The ELManager executes all Engine API requests against all attached EL nodes, in parallel. It compares their results and if there is a disagreement regarding the validity of a certain payload, this is detected and the beacon node is protected from publishing a block with a potential execution layer consensus bug in it. The BN provides metrics per EL node for the number of successful or failed requests for each type Engine API requests. If an EL node goes offline and connectivity is resoted later, we report the problem and the remedy in edge-triggered fashion. * More progress towards implementing Deneb block production in the VC and comparing the value of blocks produced by the EL and the builder API. * Adds a Makefile target for the zhejiang testnet
2023-03-05 01:40:21 +00:00
rpcClient.eth_getBlockByNumber(hexNumber, false)
func areSameAs(expectedParams: Option[NextExpectedPayloadParams],
latestHead, latestSafe, latestFinalized: Eth2Digest,
timestamp: uint64,
randomData: Eth2Digest,
feeRecipient: Eth1Address,
use fcU V3 in Deneb/Cancun (#5293)
2023-08-15 23:00:35 +00:00
withdrawals: seq[WithdrawalV1],
parentBeaconBlockRoot: FixedBytes[32]): bool =
simplify ELMonitor fcU payload attributes handling (#4696)
2023-03-06 16:19:15 +00:00
expectedParams.isSome and
expectedParams.get.headBlockHash == latestHead and
expectedParams.get.safeBlockHash == latestSafe and
expectedParams.get.finalizedBlockHash == latestFinalized and
expectedParams.get.payloadAttributes.timestamp.uint64 == timestamp and
expectedParams.get.payloadAttributes.prevRandao.bytes == randomData.data and
expectedParams.get.payloadAttributes.suggestedFeeRecipient == feeRecipient and
use fcU V3 in Deneb/Cancun (#5293)
2023-08-15 23:00:35 +00:00
expectedParams.get.payloadAttributes.withdrawals == withdrawals and
expectedParams.get.payloadAttributes.parentBeaconBlockRoot ==
parentBeaconBlockRoot
Support for driving multiple EL nodes from a single Nimbus BN (#4465) * Support for driving multiple EL nodes from a single Nimbus BN Full list of changes: * Eth1Monitor has been renamed to ELManager to match its current responsibilities better. * The ELManager is no longer optional in the code (it won't have a nil value under any circumstances). * The support for subscribing for headers was removed as it only worked with WebSockets and contributed significant complexity while bringing only a very minor advantage. * The `--web3-url` parameter has been deprecated in favor of a new `--el` parameter. The new parameter has a reasonable default value and supports specifying a different JWT for each connection. Each connection can also be configured with a different set of responsibilities (e.g. download deposits, validate blocks and/or produce blocks). On the command-line, these properties can be configured through URL properties stored in the #anchor part of the URL. In TOML files, they come with a very natural syntax (althrough the URL scheme is also supported). * The previously scattered EL-related state and logic is now moved to `eth1_monitor.nim` (this module will be renamed to `el_manager.nim` in a follow-up commit). State is assigned properly either to the `ELManager` or the to individual `ELConnection` objects where appropriate. The ELManager executes all Engine API requests against all attached EL nodes, in parallel. It compares their results and if there is a disagreement regarding the validity of a certain payload, this is detected and the beacon node is protected from publishing a block with a potential execution layer consensus bug in it. The BN provides metrics per EL node for the number of successful or failed requests for each type Engine API requests. If an EL node goes offline and connectivity is resoted later, we report the problem and the remedy in edge-triggered fashion. * More progress towards implementing Deneb block production in the VC and comparing the value of blocks produced by the EL and the builder API. * Adds a Makefile target for the zhejiang testnet
2023-03-05 01:40:21 +00:00
proc forkchoiceUpdated(rpcClient: RpcClient,
state: ForkchoiceStateV1,
always use fcUV2 in shapella even for non-proposer fcUs (#4817) * always use fcUV2 in shapella even for non-proposer fcUs * avoid template/proc naming conflict with libp2p/signed_envelope.nim having a payload proc
2023-04-17 14:17:52 +00:00
payloadAttributes: Option[PayloadAttributesV1] |
use fcU V3 in Deneb/Cancun (#5293)
2023-08-15 23:00:35 +00:00
Option[PayloadAttributesV2] |
Option[PayloadAttributesV3]):
simplify ELMonitor fcU payload attributes handling (#4696)
2023-03-06 16:19:15 +00:00
Future[ForkchoiceUpdatedResponse] =
always use fcUV2 in shapella even for non-proposer fcUs (#4817) * always use fcUV2 in shapella even for non-proposer fcUs * avoid template/proc naming conflict with libp2p/signed_envelope.nim having a payload proc
2023-04-17 14:17:52 +00:00
when payloadAttributes is Option[PayloadAttributesV1]:
rpcClient.engine_forkchoiceUpdatedV1(state, payloadAttributes)
elif payloadAttributes is Option[PayloadAttributesV2]:
rpcClient.engine_forkchoiceUpdatedV2(state, payloadAttributes)
use fcU V3 in Deneb/Cancun (#5293)
2023-08-15 23:00:35 +00:00
elif payloadAttributes is Option[PayloadAttributesV3]:
rpcClient.engine_forkchoiceUpdatedV3(state, payloadAttributes)
capella forkchoiceUpdated support (#4462) * capella forkchoiceUpdated support * match V2 fcU with V2 getPayload
2023-01-06 21:01:10 +00:00
else:
simplify ELMonitor fcU payload attributes handling (#4696)
2023-03-06 16:19:15 +00:00
static: doAssert false
Support for driving multiple EL nodes from a single Nimbus BN (#4465) * Support for driving multiple EL nodes from a single Nimbus BN Full list of changes: * Eth1Monitor has been renamed to ELManager to match its current responsibilities better. * The ELManager is no longer optional in the code (it won't have a nil value under any circumstances). * The support for subscribing for headers was removed as it only worked with WebSockets and contributed significant complexity while bringing only a very minor advantage. * The `--web3-url` parameter has been deprecated in favor of a new `--el` parameter. The new parameter has a reasonable default value and supports specifying a different JWT for each connection. Each connection can also be configured with a different set of responsibilities (e.g. download deposits, validate blocks and/or produce blocks). On the command-line, these properties can be configured through URL properties stored in the #anchor part of the URL. In TOML files, they come with a very natural syntax (althrough the URL scheme is also supported). * The previously scattered EL-related state and logic is now moved to `eth1_monitor.nim` (this module will be renamed to `el_manager.nim` in a follow-up commit). State is assigned properly either to the `ELManager` or the to individual `ELConnection` objects where appropriate. The ELManager executes all Engine API requests against all attached EL nodes, in parallel. It compares their results and if there is a disagreement regarding the validity of a certain payload, this is detected and the beacon node is protected from publishing a block with a potential execution layer consensus bug in it. The BN provides metrics per EL node for the number of successful or failed requests for each type Engine API requests. If an EL node goes offline and connectivity is resoted later, we report the problem and the remedy in edge-triggered fashion. * More progress towards implementing Deneb block production in the VC and comparing the value of blocks produced by the EL and the builder API. * Adds a Makefile target for the zhejiang testnet
2023-03-05 01:40:21 +00:00
remove `{.raises: [Defect].}` Nim 1.2 compatibility (#5352) In Nim 1.6, `{.raises: [Defect].}` is no longer necessary. Remove it.
2023-08-25 09:29:07 +00:00
func computeBlockValue(blk: ExecutionPayloadV1): UInt256 {.raises: [RlpError].} =
Support for driving multiple EL nodes from a single Nimbus BN (#4465) * Support for driving multiple EL nodes from a single Nimbus BN Full list of changes: * Eth1Monitor has been renamed to ELManager to match its current responsibilities better. * The ELManager is no longer optional in the code (it won't have a nil value under any circumstances). * The support for subscribing for headers was removed as it only worked with WebSockets and contributed significant complexity while bringing only a very minor advantage. * The `--web3-url` parameter has been deprecated in favor of a new `--el` parameter. The new parameter has a reasonable default value and supports specifying a different JWT for each connection. Each connection can also be configured with a different set of responsibilities (e.g. download deposits, validate blocks and/or produce blocks). On the command-line, these properties can be configured through URL properties stored in the #anchor part of the URL. In TOML files, they come with a very natural syntax (althrough the URL scheme is also supported). * The previously scattered EL-related state and logic is now moved to `eth1_monitor.nim` (this module will be renamed to `el_manager.nim` in a follow-up commit). State is assigned properly either to the `ELManager` or the to individual `ELConnection` objects where appropriate. The ELManager executes all Engine API requests against all attached EL nodes, in parallel. It compares their results and if there is a disagreement regarding the validity of a certain payload, this is detected and the beacon node is protected from publishing a block with a potential execution layer consensus bug in it. The BN provides metrics per EL node for the number of successful or failed requests for each type Engine API requests. If an EL node goes offline and connectivity is resoted later, we report the problem and the remedy in edge-triggered fashion. * More progress towards implementing Deneb block production in the VC and comparing the value of blocks produced by the EL and the builder API. * Adds a Makefile target for the zhejiang testnet
2023-03-05 01:40:21 +00:00
for transactionBytes in blk.transactions:
var rlp = rlpFromBytes distinctBase(transactionBytes)
let transaction = rlp.read(eth_types.Transaction)
result += distinctBase(effectiveGasTip(transaction, blk.baseFeePerGas)).u256
proc getPayloadFromSingleEL(
connection: ELConnection,
GetPayloadResponseType: type,
isForkChoiceUpToDate: bool,
use fcU V3 in Deneb/Cancun (#5293)
2023-08-15 23:00:35 +00:00
consensusHead: Eth2Digest,
Support for driving multiple EL nodes from a single Nimbus BN (#4465) * Support for driving multiple EL nodes from a single Nimbus BN Full list of changes: * Eth1Monitor has been renamed to ELManager to match its current responsibilities better. * The ELManager is no longer optional in the code (it won't have a nil value under any circumstances). * The support for subscribing for headers was removed as it only worked with WebSockets and contributed significant complexity while bringing only a very minor advantage. * The `--web3-url` parameter has been deprecated in favor of a new `--el` parameter. The new parameter has a reasonable default value and supports specifying a different JWT for each connection. Each connection can also be configured with a different set of responsibilities (e.g. download deposits, validate blocks and/or produce blocks). On the command-line, these properties can be configured through URL properties stored in the #anchor part of the URL. In TOML files, they come with a very natural syntax (althrough the URL scheme is also supported). * The previously scattered EL-related state and logic is now moved to `eth1_monitor.nim` (this module will be renamed to `el_manager.nim` in a follow-up commit). State is assigned properly either to the `ELManager` or the to individual `ELConnection` objects where appropriate. The ELManager executes all Engine API requests against all attached EL nodes, in parallel. It compares their results and if there is a disagreement regarding the validity of a certain payload, this is detected and the beacon node is protected from publishing a block with a potential execution layer consensus bug in it. The BN provides metrics per EL node for the number of successful or failed requests for each type Engine API requests. If an EL node goes offline and connectivity is resoted later, we report the problem and the remedy in edge-triggered fashion. * More progress towards implementing Deneb block production in the VC and comparing the value of blocks produced by the EL and the builder API. * Adds a Makefile target for the zhejiang testnet
2023-03-05 01:40:21 +00:00
headBlock, safeBlock, finalizedBlock: Eth2Digest,
timestamp: uint64,
randomData: Eth2Digest,
suggestedFeeRecipient: Eth1Address,
withdrawals: seq[WithdrawalV1]): Future[GetPayloadResponseType] {.async.} =
let
rpcClient = await connection.connectedRpcClient()
payloadId = if isForkChoiceUpToDate and connection.lastPayloadId.isSome:
connection.lastPayloadId.get
elif not headBlock.isZero:
engine_api_last_minute_forkchoice_updates_sent.inc(1, [connection.engineUrl.url])
simplify ELMonitor fcU payload attributes handling (#4696)
2023-03-06 16:19:15 +00:00
when GetPayloadResponseType is BellatrixExecutionPayloadWithValue:
let response = await rpcClient.forkchoiceUpdated(
ForkchoiceStateV1(
headBlockHash: headBlock.asBlockHash,
safeBlockHash: safeBlock.asBlockHash,
finalizedBlockHash: finalizedBlock.asBlockHash),
always use fcUV2 in shapella even for non-proposer fcUs (#4817) * always use fcUV2 in shapella even for non-proposer fcUs * avoid template/proc naming conflict with libp2p/signed_envelope.nim having a payload proc
2023-04-17 14:17:52 +00:00
some PayloadAttributesV1(
simplify ELMonitor fcU payload attributes handling (#4696)
2023-03-06 16:19:15 +00:00
timestamp: Quantity timestamp,
prevRandao: FixedBytes[32] randomData.data,
suggestedFeeRecipient: suggestedFeeRecipient))
use fcU V3 in Deneb/Cancun (#5293)
2023-08-15 23:00:35 +00:00
elif GetPayloadResponseType is engine_api.GetPayloadV2Response:
simplify ELMonitor fcU payload attributes handling (#4696)
2023-03-06 16:19:15 +00:00
let response = await rpcClient.forkchoiceUpdated(
ForkchoiceStateV1(
headBlockHash: headBlock.asBlockHash,
safeBlockHash: safeBlock.asBlockHash,
finalizedBlockHash: finalizedBlock.asBlockHash),
always use fcUV2 in shapella even for non-proposer fcUs (#4817) * always use fcUV2 in shapella even for non-proposer fcUs * avoid template/proc naming conflict with libp2p/signed_envelope.nim having a payload proc
2023-04-17 14:17:52 +00:00
some PayloadAttributesV2(
simplify ELMonitor fcU payload attributes handling (#4696)
2023-03-06 16:19:15 +00:00
timestamp: Quantity timestamp,
prevRandao: FixedBytes[32] randomData.data,
suggestedFeeRecipient: suggestedFeeRecipient,
withdrawals: withdrawals))
use fcU V3 in Deneb/Cancun (#5293)
2023-08-15 23:00:35 +00:00
elif GetPayloadResponseType is engine_api.GetPayloadV3Response:
let response = await rpcClient.forkchoiceUpdated(
ForkchoiceStateV1(
headBlockHash: headBlock.asBlockHash,
safeBlockHash: safeBlock.asBlockHash,
finalizedBlockHash: finalizedBlock.asBlockHash),
some PayloadAttributesV3(
timestamp: Quantity timestamp,
prevRandao: FixedBytes[32] randomData.data,
suggestedFeeRecipient: suggestedFeeRecipient,
withdrawals: withdrawals,
parentBeaconBlockRoot: consensusHead.asBlockHash))
simplify ELMonitor fcU payload attributes handling (#4696)
2023-03-06 16:19:15 +00:00
else:
static: doAssert false
Support for driving multiple EL nodes from a single Nimbus BN (#4465) * Support for driving multiple EL nodes from a single Nimbus BN Full list of changes: * Eth1Monitor has been renamed to ELManager to match its current responsibilities better. * The ELManager is no longer optional in the code (it won't have a nil value under any circumstances). * The support for subscribing for headers was removed as it only worked with WebSockets and contributed significant complexity while bringing only a very minor advantage. * The `--web3-url` parameter has been deprecated in favor of a new `--el` parameter. The new parameter has a reasonable default value and supports specifying a different JWT for each connection. Each connection can also be configured with a different set of responsibilities (e.g. download deposits, validate blocks and/or produce blocks). On the command-line, these properties can be configured through URL properties stored in the #anchor part of the URL. In TOML files, they come with a very natural syntax (althrough the URL scheme is also supported). * The previously scattered EL-related state and logic is now moved to `eth1_monitor.nim` (this module will be renamed to `el_manager.nim` in a follow-up commit). State is assigned properly either to the `ELManager` or the to individual `ELConnection` objects where appropriate. The ELManager executes all Engine API requests against all attached EL nodes, in parallel. It compares their results and if there is a disagreement regarding the validity of a certain payload, this is detected and the beacon node is protected from publishing a block with a potential execution layer consensus bug in it. The BN provides metrics per EL node for the number of successful or failed requests for each type Engine API requests. If an EL node goes offline and connectivity is resoted later, we report the problem and the remedy in edge-triggered fashion. * More progress towards implementing Deneb block production in the VC and comparing the value of blocks produced by the EL and the builder API. * Adds a Makefile target for the zhejiang testnet
2023-03-05 01:40:21 +00:00
if response.payloadStatus.status != PayloadExecutionStatus.valid or
response.payloadId.isNone:
raise newException(CatchableError, "Head block is not a valid payload")
# Give the EL some time to assemble the block
await sleepAsync(chronos.milliseconds 500)
response.payloadId.get
else:
raise newException(CatchableError, "No confirmed execution head yet")
Update nim-web3 (#4875) Update to new version of nim-web3 incorporating the execution API change that merges getPayloadV3 and getBlobsBundleV1.
2023-04-28 19:20:25 +00:00
when GetPayloadResponseType is BellatrixExecutionPayloadWithValue:
use fcU V3 in Deneb/Cancun (#5293)
2023-08-15 23:00:35 +00:00
let payload =
await engine_api.getPayload(rpcClient, ExecutionPayloadV1, payloadId)
Support for driving multiple EL nodes from a single Nimbus BN (#4465) * Support for driving multiple EL nodes from a single Nimbus BN Full list of changes: * Eth1Monitor has been renamed to ELManager to match its current responsibilities better. * The ELManager is no longer optional in the code (it won't have a nil value under any circumstances). * The support for subscribing for headers was removed as it only worked with WebSockets and contributed significant complexity while bringing only a very minor advantage. * The `--web3-url` parameter has been deprecated in favor of a new `--el` parameter. The new parameter has a reasonable default value and supports specifying a different JWT for each connection. Each connection can also be configured with a different set of responsibilities (e.g. download deposits, validate blocks and/or produce blocks). On the command-line, these properties can be configured through URL properties stored in the #anchor part of the URL. In TOML files, they come with a very natural syntax (althrough the URL scheme is also supported). * The previously scattered EL-related state and logic is now moved to `eth1_monitor.nim` (this module will be renamed to `el_manager.nim` in a follow-up commit). State is assigned properly either to the `ELManager` or the to individual `ELConnection` objects where appropriate. The ELManager executes all Engine API requests against all attached EL nodes, in parallel. It compares their results and if there is a disagreement regarding the validity of a certain payload, this is detected and the beacon node is protected from publishing a block with a potential execution layer consensus bug in it. The BN provides metrics per EL node for the number of successful or failed requests for each type Engine API requests. If an EL node goes offline and connectivity is resoted later, we report the problem and the remedy in edge-triggered fashion. * More progress towards implementing Deneb block production in the VC and comparing the value of blocks produced by the EL and the builder API. * Adds a Makefile target for the zhejiang testnet
2023-03-05 01:40:21 +00:00
return BellatrixExecutionPayloadWithValue(
executionPayload: payload,
blockValue: computeBlockValue payload)
else:
return await engine_api.getPayload(rpcClient, GetPayloadResponseType, payloadId)
simplify ELMonitor fcU payload attributes handling (#4696)
2023-03-06 16:19:15 +00:00
func cmpGetPayloadResponses(lhs, rhs: SomeEnginePayloadWithValue): int =
Support for driving multiple EL nodes from a single Nimbus BN (#4465) * Support for driving multiple EL nodes from a single Nimbus BN Full list of changes: * Eth1Monitor has been renamed to ELManager to match its current responsibilities better. * The ELManager is no longer optional in the code (it won't have a nil value under any circumstances). * The support for subscribing for headers was removed as it only worked with WebSockets and contributed significant complexity while bringing only a very minor advantage. * The `--web3-url` parameter has been deprecated in favor of a new `--el` parameter. The new parameter has a reasonable default value and supports specifying a different JWT for each connection. Each connection can also be configured with a different set of responsibilities (e.g. download deposits, validate blocks and/or produce blocks). On the command-line, these properties can be configured through URL properties stored in the #anchor part of the URL. In TOML files, they come with a very natural syntax (althrough the URL scheme is also supported). * The previously scattered EL-related state and logic is now moved to `eth1_monitor.nim` (this module will be renamed to `el_manager.nim` in a follow-up commit). State is assigned properly either to the `ELManager` or the to individual `ELConnection` objects where appropriate. The ELManager executes all Engine API requests against all attached EL nodes, in parallel. It compares their results and if there is a disagreement regarding the validity of a certain payload, this is detected and the beacon node is protected from publishing a block with a potential execution layer consensus bug in it. The BN provides metrics per EL node for the number of successful or failed requests for each type Engine API requests. If an EL node goes offline and connectivity is resoted later, we report the problem and the remedy in edge-triggered fashion. * More progress towards implementing Deneb block production in the VC and comparing the value of blocks produced by the EL and the builder API. * Adds a Makefile target for the zhejiang testnet
2023-03-05 01:40:21 +00:00
cmp(distinctBase lhs.blockValue, distinctBase rhs.blockValue)
partially integrate eth1 merge changes (#2548) * partially integrate eth1 merge changes * use hexToSeqByte() and validate execution engine opaque transaction length * remove incorrect REST serialization code
2021-05-20 10:44:13 +00:00
Support for driving multiple EL nodes from a single Nimbus BN (#4465) * Support for driving multiple EL nodes from a single Nimbus BN Full list of changes: * Eth1Monitor has been renamed to ELManager to match its current responsibilities better. * The ELManager is no longer optional in the code (it won't have a nil value under any circumstances). * The support for subscribing for headers was removed as it only worked with WebSockets and contributed significant complexity while bringing only a very minor advantage. * The `--web3-url` parameter has been deprecated in favor of a new `--el` parameter. The new parameter has a reasonable default value and supports specifying a different JWT for each connection. Each connection can also be configured with a different set of responsibilities (e.g. download deposits, validate blocks and/or produce blocks). On the command-line, these properties can be configured through URL properties stored in the #anchor part of the URL. In TOML files, they come with a very natural syntax (althrough the URL scheme is also supported). * The previously scattered EL-related state and logic is now moved to `eth1_monitor.nim` (this module will be renamed to `el_manager.nim` in a follow-up commit). State is assigned properly either to the `ELManager` or the to individual `ELConnection` objects where appropriate. The ELManager executes all Engine API requests against all attached EL nodes, in parallel. It compares their results and if there is a disagreement regarding the validity of a certain payload, this is detected and the beacon node is protected from publishing a block with a potential execution layer consensus bug in it. The BN provides metrics per EL node for the number of successful or failed requests for each type Engine API requests. If an EL node goes offline and connectivity is resoted later, we report the problem and the remedy in edge-triggered fashion. * More progress towards implementing Deneb block production in the VC and comparing the value of blocks produced by the EL and the builder API. * Adds a Makefile target for the zhejiang testnet
2023-03-05 01:40:21 +00:00
template EngineApiResponseType*(T: type bellatrix.ExecutionPayloadForSigning): type =
BellatrixExecutionPayloadWithValue
template EngineApiResponseType*(T: type capella.ExecutionPayloadForSigning): type =
engine_api.GetPayloadV2Response
template EngineApiResponseType*(T: type deneb.ExecutionPayloadForSigning): type =
Update nim-web3 (#4875) Update to new version of nim-web3 incorporating the execution API change that merges getPayloadV3 and getBlobsBundleV1.
2023-04-28 19:20:25 +00:00
engine_api.GetPayloadV3Response
Support for driving multiple EL nodes from a single Nimbus BN (#4465) * Support for driving multiple EL nodes from a single Nimbus BN Full list of changes: * Eth1Monitor has been renamed to ELManager to match its current responsibilities better. * The ELManager is no longer optional in the code (it won't have a nil value under any circumstances). * The support for subscribing for headers was removed as it only worked with WebSockets and contributed significant complexity while bringing only a very minor advantage. * The `--web3-url` parameter has been deprecated in favor of a new `--el` parameter. The new parameter has a reasonable default value and supports specifying a different JWT for each connection. Each connection can also be configured with a different set of responsibilities (e.g. download deposits, validate blocks and/or produce blocks). On the command-line, these properties can be configured through URL properties stored in the #anchor part of the URL. In TOML files, they come with a very natural syntax (althrough the URL scheme is also supported). * The previously scattered EL-related state and logic is now moved to `eth1_monitor.nim` (this module will be renamed to `el_manager.nim` in a follow-up commit). State is assigned properly either to the `ELManager` or the to individual `ELConnection` objects where appropriate. The ELManager executes all Engine API requests against all attached EL nodes, in parallel. It compares their results and if there is a disagreement regarding the validity of a certain payload, this is detected and the beacon node is protected from publishing a block with a potential execution layer consensus bug in it. The BN provides metrics per EL node for the number of successful or failed requests for each type Engine API requests. If an EL node goes offline and connectivity is resoted later, we report the problem and the remedy in edge-triggered fashion. * More progress towards implementing Deneb block production in the VC and comparing the value of blocks produced by the EL and the builder API. * Adds a Makefile target for the zhejiang testnet
2023-03-05 01:40:21 +00:00
template payload(response: engine_api.ExecutionPayloadV1): engine_api.ExecutionPayloadV1 =
response
template payload(response: engine_api.GetPayloadV2Response): engine_api.ExecutionPayloadV1OrV2 =
response.executionPayload
template payload(response: engine_api.GetPayloadV3Response): engine_api.ExecutionPayloadV3 =
response.executionPayload
template toEngineWithdrawals*(withdrawals: seq[capella.Withdrawal]): seq[WithdrawalV1] =
mapIt(withdrawals, toEngineWithdrawal(it))
convert remaining `toFork` -> `kind` for consistency (#5462) We currently have multiple ways to obtain `ConsensusFork` or `LcDataFork` from a forky type. Rename `toFork` to `kind` for a consistent API naming.
2023-09-27 15:10:28 +00:00
template kind(T: type ExecutionPayloadV1): ConsensusFork =
Support for driving multiple EL nodes from a single Nimbus BN (#4465) * Support for driving multiple EL nodes from a single Nimbus BN Full list of changes: * Eth1Monitor has been renamed to ELManager to match its current responsibilities better. * The ELManager is no longer optional in the code (it won't have a nil value under any circumstances). * The support for subscribing for headers was removed as it only worked with WebSockets and contributed significant complexity while bringing only a very minor advantage. * The `--web3-url` parameter has been deprecated in favor of a new `--el` parameter. The new parameter has a reasonable default value and supports specifying a different JWT for each connection. Each connection can also be configured with a different set of responsibilities (e.g. download deposits, validate blocks and/or produce blocks). On the command-line, these properties can be configured through URL properties stored in the #anchor part of the URL. In TOML files, they come with a very natural syntax (althrough the URL scheme is also supported). * The previously scattered EL-related state and logic is now moved to `eth1_monitor.nim` (this module will be renamed to `el_manager.nim` in a follow-up commit). State is assigned properly either to the `ELManager` or the to individual `ELConnection` objects where appropriate. The ELManager executes all Engine API requests against all attached EL nodes, in parallel. It compares their results and if there is a disagreement regarding the validity of a certain payload, this is detected and the beacon node is protected from publishing a block with a potential execution layer consensus bug in it. The BN provides metrics per EL node for the number of successful or failed requests for each type Engine API requests. If an EL node goes offline and connectivity is resoted later, we report the problem and the remedy in edge-triggered fashion. * More progress towards implementing Deneb block production in the VC and comparing the value of blocks produced by the EL and the builder API. * Adds a Makefile target for the zhejiang testnet
2023-03-05 01:40:21 +00:00
ConsensusFork.Bellatrix
convert remaining `toFork` -> `kind` for consistency (#5462) We currently have multiple ways to obtain `ConsensusFork` or `LcDataFork` from a forky type. Rename `toFork` to `kind` for a consistent API naming.
2023-09-27 15:10:28 +00:00
template kind(T: typedesc[ExecutionPayloadV1OrV2|ExecutionPayloadV2]): ConsensusFork =
Support for driving multiple EL nodes from a single Nimbus BN (#4465) * Support for driving multiple EL nodes from a single Nimbus BN Full list of changes: * Eth1Monitor has been renamed to ELManager to match its current responsibilities better. * The ELManager is no longer optional in the code (it won't have a nil value under any circumstances). * The support for subscribing for headers was removed as it only worked with WebSockets and contributed significant complexity while bringing only a very minor advantage. * The `--web3-url` parameter has been deprecated in favor of a new `--el` parameter. The new parameter has a reasonable default value and supports specifying a different JWT for each connection. Each connection can also be configured with a different set of responsibilities (e.g. download deposits, validate blocks and/or produce blocks). On the command-line, these properties can be configured through URL properties stored in the #anchor part of the URL. In TOML files, they come with a very natural syntax (althrough the URL scheme is also supported). * The previously scattered EL-related state and logic is now moved to `eth1_monitor.nim` (this module will be renamed to `el_manager.nim` in a follow-up commit). State is assigned properly either to the `ELManager` or the to individual `ELConnection` objects where appropriate. The ELManager executes all Engine API requests against all attached EL nodes, in parallel. It compares their results and if there is a disagreement regarding the validity of a certain payload, this is detected and the beacon node is protected from publishing a block with a potential execution layer consensus bug in it. The BN provides metrics per EL node for the number of successful or failed requests for each type Engine API requests. If an EL node goes offline and connectivity is resoted later, we report the problem and the remedy in edge-triggered fashion. * More progress towards implementing Deneb block production in the VC and comparing the value of blocks produced by the EL and the builder API. * Adds a Makefile target for the zhejiang testnet
2023-03-05 01:40:21 +00:00
ConsensusFork.Capella
convert remaining `toFork` -> `kind` for consistency (#5462) We currently have multiple ways to obtain `ConsensusFork` or `LcDataFork` from a forky type. Rename `toFork` to `kind` for a consistent API naming.
2023-09-27 15:10:28 +00:00
template kind(T: type ExecutionPayloadV3): ConsensusFork =
Support for driving multiple EL nodes from a single Nimbus BN (#4465) * Support for driving multiple EL nodes from a single Nimbus BN Full list of changes: * Eth1Monitor has been renamed to ELManager to match its current responsibilities better. * The ELManager is no longer optional in the code (it won't have a nil value under any circumstances). * The support for subscribing for headers was removed as it only worked with WebSockets and contributed significant complexity while bringing only a very minor advantage. * The `--web3-url` parameter has been deprecated in favor of a new `--el` parameter. The new parameter has a reasonable default value and supports specifying a different JWT for each connection. Each connection can also be configured with a different set of responsibilities (e.g. download deposits, validate blocks and/or produce blocks). On the command-line, these properties can be configured through URL properties stored in the #anchor part of the URL. In TOML files, they come with a very natural syntax (althrough the URL scheme is also supported). * The previously scattered EL-related state and logic is now moved to `eth1_monitor.nim` (this module will be renamed to `el_manager.nim` in a follow-up commit). State is assigned properly either to the `ELManager` or the to individual `ELConnection` objects where appropriate. The ELManager executes all Engine API requests against all attached EL nodes, in parallel. It compares their results and if there is a disagreement regarding the validity of a certain payload, this is detected and the beacon node is protected from publishing a block with a potential execution layer consensus bug in it. The BN provides metrics per EL node for the number of successful or failed requests for each type Engine API requests. If an EL node goes offline and connectivity is resoted later, we report the problem and the remedy in edge-triggered fashion. * More progress towards implementing Deneb block production in the VC and comparing the value of blocks produced by the EL and the builder API. * Adds a Makefile target for the zhejiang testnet
2023-03-05 01:40:21 +00:00
ConsensusFork.Deneb
proc getPayload*(m: ELManager,
PayloadType: type ForkyExecutionPayloadForSigning,
use fcU V3 in Deneb/Cancun (#5293)
2023-08-15 23:00:35 +00:00
consensusHead: Eth2Digest,
Support for driving multiple EL nodes from a single Nimbus BN (#4465) * Support for driving multiple EL nodes from a single Nimbus BN Full list of changes: * Eth1Monitor has been renamed to ELManager to match its current responsibilities better. * The ELManager is no longer optional in the code (it won't have a nil value under any circumstances). * The support for subscribing for headers was removed as it only worked with WebSockets and contributed significant complexity while bringing only a very minor advantage. * The `--web3-url` parameter has been deprecated in favor of a new `--el` parameter. The new parameter has a reasonable default value and supports specifying a different JWT for each connection. Each connection can also be configured with a different set of responsibilities (e.g. download deposits, validate blocks and/or produce blocks). On the command-line, these properties can be configured through URL properties stored in the #anchor part of the URL. In TOML files, they come with a very natural syntax (althrough the URL scheme is also supported). * The previously scattered EL-related state and logic is now moved to `eth1_monitor.nim` (this module will be renamed to `el_manager.nim` in a follow-up commit). State is assigned properly either to the `ELManager` or the to individual `ELConnection` objects where appropriate. The ELManager executes all Engine API requests against all attached EL nodes, in parallel. It compares their results and if there is a disagreement regarding the validity of a certain payload, this is detected and the beacon node is protected from publishing a block with a potential execution layer consensus bug in it. The BN provides metrics per EL node for the number of successful or failed requests for each type Engine API requests. If an EL node goes offline and connectivity is resoted later, we report the problem and the remedy in edge-triggered fashion. * More progress towards implementing Deneb block production in the VC and comparing the value of blocks produced by the EL and the builder API. * Adds a Makefile target for the zhejiang testnet
2023-03-05 01:40:21 +00:00
headBlock, safeBlock, finalizedBlock: Eth2Digest,
timestamp: uint64,
randomData: Eth2Digest,
suggestedFeeRecipient: Eth1Address,
withdrawals: seq[capella.Withdrawal]):
Future[Opt[PayloadType]] {.async.} =
if m.elConnections.len == 0:
return err()
let
engineApiWithdrawals = toEngineWithdrawals withdrawals
use fcU V3 in Deneb/Cancun (#5293)
2023-08-15 23:00:35 +00:00
isFcUpToDate = m.nextExpectedPayloadParams.areSameAs(
headBlock, safeBlock, finalizedBlock, timestamp,
randomData, suggestedFeeRecipient, engineApiWithdrawals,
consensusHead.asBlockHash)
Support for driving multiple EL nodes from a single Nimbus BN (#4465) * Support for driving multiple EL nodes from a single Nimbus BN Full list of changes: * Eth1Monitor has been renamed to ELManager to match its current responsibilities better. * The ELManager is no longer optional in the code (it won't have a nil value under any circumstances). * The support for subscribing for headers was removed as it only worked with WebSockets and contributed significant complexity while bringing only a very minor advantage. * The `--web3-url` parameter has been deprecated in favor of a new `--el` parameter. The new parameter has a reasonable default value and supports specifying a different JWT for each connection. Each connection can also be configured with a different set of responsibilities (e.g. download deposits, validate blocks and/or produce blocks). On the command-line, these properties can be configured through URL properties stored in the #anchor part of the URL. In TOML files, they come with a very natural syntax (althrough the URL scheme is also supported). * The previously scattered EL-related state and logic is now moved to `eth1_monitor.nim` (this module will be renamed to `el_manager.nim` in a follow-up commit). State is assigned properly either to the `ELManager` or the to individual `ELConnection` objects where appropriate. The ELManager executes all Engine API requests against all attached EL nodes, in parallel. It compares their results and if there is a disagreement regarding the validity of a certain payload, this is detected and the beacon node is protected from publishing a block with a potential execution layer consensus bug in it. The BN provides metrics per EL node for the number of successful or failed requests for each type Engine API requests. If an EL node goes offline and connectivity is resoted later, we report the problem and the remedy in edge-triggered fashion. * More progress towards implementing Deneb block production in the VC and comparing the value of blocks produced by the EL and the builder API. * Adds a Makefile target for the zhejiang testnet
2023-03-05 01:40:21 +00:00
increase `getPayloadFromSingleEL` timeout (#5194) When producing a local block and `runProposalForkchoiceUpdated` was missed, `getPayloadFromSingleEL` adds additional ~500ms of latency. Quick fix to avoid missing blocks to that.
2023-07-18 20:29:50 +00:00
# `getPayloadFromSingleEL` may introduce additional latency
const extraProcessingOverhead = 500.milliseconds
Support for driving multiple EL nodes from a single Nimbus BN (#4465) * Support for driving multiple EL nodes from a single Nimbus BN Full list of changes: * Eth1Monitor has been renamed to ELManager to match its current responsibilities better. * The ELManager is no longer optional in the code (it won't have a nil value under any circumstances). * The support for subscribing for headers was removed as it only worked with WebSockets and contributed significant complexity while bringing only a very minor advantage. * The `--web3-url` parameter has been deprecated in favor of a new `--el` parameter. The new parameter has a reasonable default value and supports specifying a different JWT for each connection. Each connection can also be configured with a different set of responsibilities (e.g. download deposits, validate blocks and/or produce blocks). On the command-line, these properties can be configured through URL properties stored in the #anchor part of the URL. In TOML files, they come with a very natural syntax (althrough the URL scheme is also supported). * The previously scattered EL-related state and logic is now moved to `eth1_monitor.nim` (this module will be renamed to `el_manager.nim` in a follow-up commit). State is assigned properly either to the `ELManager` or the to individual `ELConnection` objects where appropriate. The ELManager executes all Engine API requests against all attached EL nodes, in parallel. It compares their results and if there is a disagreement regarding the validity of a certain payload, this is detected and the beacon node is protected from publishing a block with a potential execution layer consensus bug in it. The BN provides metrics per EL node for the number of successful or failed requests for each type Engine API requests. If an EL node goes offline and connectivity is resoted later, we report the problem and the remedy in edge-triggered fashion. * More progress towards implementing Deneb block production in the VC and comparing the value of blocks produced by the EL and the builder API. * Adds a Makefile target for the zhejiang testnet
2023-03-05 01:40:21 +00:00
let
increase `getPayloadFromSingleEL` timeout (#5194) When producing a local block and `runProposalForkchoiceUpdated` was missed, `getPayloadFromSingleEL` adds additional ~500ms of latency. Quick fix to avoid missing blocks to that.
2023-07-18 20:29:50 +00:00
timeout = GETPAYLOAD_TIMEOUT + extraProcessingOverhead
Support for driving multiple EL nodes from a single Nimbus BN (#4465) * Support for driving multiple EL nodes from a single Nimbus BN Full list of changes: * Eth1Monitor has been renamed to ELManager to match its current responsibilities better. * The ELManager is no longer optional in the code (it won't have a nil value under any circumstances). * The support for subscribing for headers was removed as it only worked with WebSockets and contributed significant complexity while bringing only a very minor advantage. * The `--web3-url` parameter has been deprecated in favor of a new `--el` parameter. The new parameter has a reasonable default value and supports specifying a different JWT for each connection. Each connection can also be configured with a different set of responsibilities (e.g. download deposits, validate blocks and/or produce blocks). On the command-line, these properties can be configured through URL properties stored in the #anchor part of the URL. In TOML files, they come with a very natural syntax (althrough the URL scheme is also supported). * The previously scattered EL-related state and logic is now moved to `eth1_monitor.nim` (this module will be renamed to `el_manager.nim` in a follow-up commit). State is assigned properly either to the `ELManager` or the to individual `ELConnection` objects where appropriate. The ELManager executes all Engine API requests against all attached EL nodes, in parallel. It compares their results and if there is a disagreement regarding the validity of a certain payload, this is detected and the beacon node is protected from publishing a block with a potential execution layer consensus bug in it. The BN provides metrics per EL node for the number of successful or failed requests for each type Engine API requests. If an EL node goes offline and connectivity is resoted later, we report the problem and the remedy in edge-triggered fashion. * More progress towards implementing Deneb block production in the VC and comparing the value of blocks produced by the EL and the builder API. * Adds a Makefile target for the zhejiang testnet
2023-03-05 01:40:21 +00:00
deadline = sleepAsync(timeout)
requests = m.elConnections.mapIt(it.getPayloadFromSingleEL(
EngineApiResponseType(PayloadType),
use fcU V3 in Deneb/Cancun (#5293)
2023-08-15 23:00:35 +00:00
isFcUpToDate, consensusHead, headBlock, safeBlock, finalizedBlock,
Support for driving multiple EL nodes from a single Nimbus BN (#4465) * Support for driving multiple EL nodes from a single Nimbus BN Full list of changes: * Eth1Monitor has been renamed to ELManager to match its current responsibilities better. * The ELManager is no longer optional in the code (it won't have a nil value under any circumstances). * The support for subscribing for headers was removed as it only worked with WebSockets and contributed significant complexity while bringing only a very minor advantage. * The `--web3-url` parameter has been deprecated in favor of a new `--el` parameter. The new parameter has a reasonable default value and supports specifying a different JWT for each connection. Each connection can also be configured with a different set of responsibilities (e.g. download deposits, validate blocks and/or produce blocks). On the command-line, these properties can be configured through URL properties stored in the #anchor part of the URL. In TOML files, they come with a very natural syntax (althrough the URL scheme is also supported). * The previously scattered EL-related state and logic is now moved to `eth1_monitor.nim` (this module will be renamed to `el_manager.nim` in a follow-up commit). State is assigned properly either to the `ELManager` or the to individual `ELConnection` objects where appropriate. The ELManager executes all Engine API requests against all attached EL nodes, in parallel. It compares their results and if there is a disagreement regarding the validity of a certain payload, this is detected and the beacon node is protected from publishing a block with a potential execution layer consensus bug in it. The BN provides metrics per EL node for the number of successful or failed requests for each type Engine API requests. If an EL node goes offline and connectivity is resoted later, we report the problem and the remedy in edge-triggered fashion. * More progress towards implementing Deneb block production in the VC and comparing the value of blocks produced by the EL and the builder API. * Adds a Makefile target for the zhejiang testnet
2023-03-05 01:40:21 +00:00
timestamp, randomData, suggestedFeeRecipient, engineApiWithdrawals
))
requestsCompleted = allFutures(requests)
await requestsCompleted or deadline
var bestPayloadIdx = none int
for idx, req in requests:
if not req.finished:
BN+LC+SN: Fix cancellation deprecate warnings. (#5455) * Fix deprecation cancel() warnings in SN, BN, LC. * Fix missing import.
2023-09-22 11:06:27 +00:00
req.cancelSoon()
Support for driving multiple EL nodes from a single Nimbus BN (#4465) * Support for driving multiple EL nodes from a single Nimbus BN Full list of changes: * Eth1Monitor has been renamed to ELManager to match its current responsibilities better. * The ELManager is no longer optional in the code (it won't have a nil value under any circumstances). * The support for subscribing for headers was removed as it only worked with WebSockets and contributed significant complexity while bringing only a very minor advantage. * The `--web3-url` parameter has been deprecated in favor of a new `--el` parameter. The new parameter has a reasonable default value and supports specifying a different JWT for each connection. Each connection can also be configured with a different set of responsibilities (e.g. download deposits, validate blocks and/or produce blocks). On the command-line, these properties can be configured through URL properties stored in the #anchor part of the URL. In TOML files, they come with a very natural syntax (althrough the URL scheme is also supported). * The previously scattered EL-related state and logic is now moved to `eth1_monitor.nim` (this module will be renamed to `el_manager.nim` in a follow-up commit). State is assigned properly either to the `ELManager` or the to individual `ELConnection` objects where appropriate. The ELManager executes all Engine API requests against all attached EL nodes, in parallel. It compares their results and if there is a disagreement regarding the validity of a certain payload, this is detected and the beacon node is protected from publishing a block with a potential execution layer consensus bug in it. The BN provides metrics per EL node for the number of successful or failed requests for each type Engine API requests. If an EL node goes offline and connectivity is resoted later, we report the problem and the remedy in edge-triggered fashion. * More progress towards implementing Deneb block production in the VC and comparing the value of blocks produced by the EL and the builder API. * Adds a Makefile target for the zhejiang testnet
2023-03-05 01:40:21 +00:00
elif req.failed:
error "Failed to get execution payload from EL",
Cosmetic logging changes in the EL manager
2023-03-10 09:12:29 +00:00
url = m.elConnections[idx].engineUrl.url,
err = req.error.msg
Support for driving multiple EL nodes from a single Nimbus BN (#4465) * Support for driving multiple EL nodes from a single Nimbus BN Full list of changes: * Eth1Monitor has been renamed to ELManager to match its current responsibilities better. * The ELManager is no longer optional in the code (it won't have a nil value under any circumstances). * The support for subscribing for headers was removed as it only worked with WebSockets and contributed significant complexity while bringing only a very minor advantage. * The `--web3-url` parameter has been deprecated in favor of a new `--el` parameter. The new parameter has a reasonable default value and supports specifying a different JWT for each connection. Each connection can also be configured with a different set of responsibilities (e.g. download deposits, validate blocks and/or produce blocks). On the command-line, these properties can be configured through URL properties stored in the #anchor part of the URL. In TOML files, they come with a very natural syntax (althrough the URL scheme is also supported). * The previously scattered EL-related state and logic is now moved to `eth1_monitor.nim` (this module will be renamed to `el_manager.nim` in a follow-up commit). State is assigned properly either to the `ELManager` or the to individual `ELConnection` objects where appropriate. The ELManager executes all Engine API requests against all attached EL nodes, in parallel. It compares their results and if there is a disagreement regarding the validity of a certain payload, this is detected and the beacon node is protected from publishing a block with a potential execution layer consensus bug in it. The BN provides metrics per EL node for the number of successful or failed requests for each type Engine API requests. If an EL node goes offline and connectivity is resoted later, we report the problem and the remedy in edge-triggered fashion. * More progress towards implementing Deneb block production in the VC and comparing the value of blocks produced by the EL and the builder API. * Adds a Makefile target for the zhejiang testnet
2023-03-05 01:40:21 +00:00
else:
convert remaining `toFork` -> `kind` for consistency (#5462) We currently have multiple ways to obtain `ConsensusFork` or `LcDataFork` from a forky type. Rename `toFork` to `kind` for a consistent API naming.
2023-09-27 15:10:28 +00:00
const payloadFork = PayloadType.kind
Support for driving multiple EL nodes from a single Nimbus BN (#4465) * Support for driving multiple EL nodes from a single Nimbus BN Full list of changes: * Eth1Monitor has been renamed to ELManager to match its current responsibilities better. * The ELManager is no longer optional in the code (it won't have a nil value under any circumstances). * The support for subscribing for headers was removed as it only worked with WebSockets and contributed significant complexity while bringing only a very minor advantage. * The `--web3-url` parameter has been deprecated in favor of a new `--el` parameter. The new parameter has a reasonable default value and supports specifying a different JWT for each connection. Each connection can also be configured with a different set of responsibilities (e.g. download deposits, validate blocks and/or produce blocks). On the command-line, these properties can be configured through URL properties stored in the #anchor part of the URL. In TOML files, they come with a very natural syntax (althrough the URL scheme is also supported). * The previously scattered EL-related state and logic is now moved to `eth1_monitor.nim` (this module will be renamed to `el_manager.nim` in a follow-up commit). State is assigned properly either to the `ELManager` or the to individual `ELConnection` objects where appropriate. The ELManager executes all Engine API requests against all attached EL nodes, in parallel. It compares their results and if there is a disagreement regarding the validity of a certain payload, this is detected and the beacon node is protected from publishing a block with a potential execution layer consensus bug in it. The BN provides metrics per EL node for the number of successful or failed requests for each type Engine API requests. If an EL node goes offline and connectivity is resoted later, we report the problem and the remedy in edge-triggered fashion. * More progress towards implementing Deneb block production in the VC and comparing the value of blocks produced by the EL and the builder API. * Adds a Makefile target for the zhejiang testnet
2023-03-05 01:40:21 +00:00
when payloadFork >= ConsensusFork.Capella:
when payloadFork == ConsensusFork.Capella:
# TODO: The engine_api module may offer an alternative API where it is guaranteed
# to return the correct response type (i.e. the rule below will be enforced
# during deserialization).
if req.read.executionPayload.withdrawals.isNone:
Cosmetic logging changes in the EL manager
2023-03-10 09:12:29 +00:00
warn "Execution client returned a block without a 'withdrawals' field for a post-Shanghai block",
Support for driving multiple EL nodes from a single Nimbus BN (#4465) * Support for driving multiple EL nodes from a single Nimbus BN Full list of changes: * Eth1Monitor has been renamed to ELManager to match its current responsibilities better. * The ELManager is no longer optional in the code (it won't have a nil value under any circumstances). * The support for subscribing for headers was removed as it only worked with WebSockets and contributed significant complexity while bringing only a very minor advantage. * The `--web3-url` parameter has been deprecated in favor of a new `--el` parameter. The new parameter has a reasonable default value and supports specifying a different JWT for each connection. Each connection can also be configured with a different set of responsibilities (e.g. download deposits, validate blocks and/or produce blocks). On the command-line, these properties can be configured through URL properties stored in the #anchor part of the URL. In TOML files, they come with a very natural syntax (althrough the URL scheme is also supported). * The previously scattered EL-related state and logic is now moved to `eth1_monitor.nim` (this module will be renamed to `el_manager.nim` in a follow-up commit). State is assigned properly either to the `ELManager` or the to individual `ELConnection` objects where appropriate. The ELManager executes all Engine API requests against all attached EL nodes, in parallel. It compares their results and if there is a disagreement regarding the validity of a certain payload, this is detected and the beacon node is protected from publishing a block with a potential execution layer consensus bug in it. The BN provides metrics per EL node for the number of successful or failed requests for each type Engine API requests. If an EL node goes offline and connectivity is resoted later, we report the problem and the remedy in edge-triggered fashion. * More progress towards implementing Deneb block production in the VC and comparing the value of blocks produced by the EL and the builder API. * Adds a Makefile target for the zhejiang testnet
2023-03-05 01:40:21 +00:00
url = m.elConnections[idx].engineUrl.url
continue
if engineApiWithdrawals != req.read.executionPayload.withdrawals.maybeDeref:
fix expected withdrawal mismatch logging (#4798)
2023-04-08 09:15:09 +00:00
# otherwise it formats as "@[(index: ..., validatorIndex: ...,
# address: ..., amount: ...), (index: ..., validatorIndex: ...,
# address: ..., amount: ...)]"
Support for driving multiple EL nodes from a single Nimbus BN (#4465) * Support for driving multiple EL nodes from a single Nimbus BN Full list of changes: * Eth1Monitor has been renamed to ELManager to match its current responsibilities better. * The ELManager is no longer optional in the code (it won't have a nil value under any circumstances). * The support for subscribing for headers was removed as it only worked with WebSockets and contributed significant complexity while bringing only a very minor advantage. * The `--web3-url` parameter has been deprecated in favor of a new `--el` parameter. The new parameter has a reasonable default value and supports specifying a different JWT for each connection. Each connection can also be configured with a different set of responsibilities (e.g. download deposits, validate blocks and/or produce blocks). On the command-line, these properties can be configured through URL properties stored in the #anchor part of the URL. In TOML files, they come with a very natural syntax (althrough the URL scheme is also supported). * The previously scattered EL-related state and logic is now moved to `eth1_monitor.nim` (this module will be renamed to `el_manager.nim` in a follow-up commit). State is assigned properly either to the `ELManager` or the to individual `ELConnection` objects where appropriate. The ELManager executes all Engine API requests against all attached EL nodes, in parallel. It compares their results and if there is a disagreement regarding the validity of a certain payload, this is detected and the beacon node is protected from publishing a block with a potential execution layer consensus bug in it. The BN provides metrics per EL node for the number of successful or failed requests for each type Engine API requests. If an EL node goes offline and connectivity is resoted later, we report the problem and the remedy in edge-triggered fashion. * More progress towards implementing Deneb block production in the VC and comparing the value of blocks produced by the EL and the builder API. * Adds a Makefile target for the zhejiang testnet
2023-03-05 01:40:21 +00:00
warn "Execution client did not return correct withdrawals",
fix expected withdrawal mismatch logging (#4798)
2023-04-08 09:15:09 +00:00
withdrawals_from_cl_len = engineApiWithdrawals.len,
withdrawals_from_el_len =
req.read.executionPayload.withdrawals.maybeDeref.len,
withdrawals_from_cl =
mapIt(engineApiWithdrawals, it.asConsensusWithdrawal),
withdrawals_from_el =
mapIt(
req.read.executionPayload.withdrawals.maybeDeref,
it.asConsensusWithdrawal)
Support for driving multiple EL nodes from a single Nimbus BN (#4465) * Support for driving multiple EL nodes from a single Nimbus BN Full list of changes: * Eth1Monitor has been renamed to ELManager to match its current responsibilities better. * The ELManager is no longer optional in the code (it won't have a nil value under any circumstances). * The support for subscribing for headers was removed as it only worked with WebSockets and contributed significant complexity while bringing only a very minor advantage. * The `--web3-url` parameter has been deprecated in favor of a new `--el` parameter. The new parameter has a reasonable default value and supports specifying a different JWT for each connection. Each connection can also be configured with a different set of responsibilities (e.g. download deposits, validate blocks and/or produce blocks). On the command-line, these properties can be configured through URL properties stored in the #anchor part of the URL. In TOML files, they come with a very natural syntax (althrough the URL scheme is also supported). * The previously scattered EL-related state and logic is now moved to `eth1_monitor.nim` (this module will be renamed to `el_manager.nim` in a follow-up commit). State is assigned properly either to the `ELManager` or the to individual `ELConnection` objects where appropriate. The ELManager executes all Engine API requests against all attached EL nodes, in parallel. It compares their results and if there is a disagreement regarding the validity of a certain payload, this is detected and the beacon node is protected from publishing a block with a potential execution layer consensus bug in it. The BN provides metrics per EL node for the number of successful or failed requests for each type Engine API requests. If an EL node goes offline and connectivity is resoted later, we report the problem and the remedy in edge-triggered fashion. * More progress towards implementing Deneb block production in the VC and comparing the value of blocks produced by the EL and the builder API. * Adds a Makefile target for the zhejiang testnet
2023-03-05 01:40:21 +00:00
Bugfix: historic blocks violate the latest specs for the extraData field (fixes #4695) (#4700)
2023-03-07 04:43:27 +00:00
if req.read.executionPayload.extraData.len > MAX_EXTRA_DATA_BYTES:
warn "Execution client provided a block with invalid extraData (size exceeds limit)",
More metrics for the EL monitor (#4707) * `engine_api_response_time` provides a histogram for the Engine API response times for each unique pair ot URL and request type. * All engine API requests are now tracked Other changes: The client will no longer exit on start-up if it fails to connect to a properly configured EL node.
2023-03-09 17:29:04 +00:00
size = req.read.executionPayload.extraData.len,
limit = MAX_EXTRA_DATA_BYTES
Bugfix: historic blocks violate the latest specs for the extraData field (fixes #4695) (#4700)
2023-03-07 04:43:27 +00:00
continue
Support for driving multiple EL nodes from a single Nimbus BN (#4465) * Support for driving multiple EL nodes from a single Nimbus BN Full list of changes: * Eth1Monitor has been renamed to ELManager to match its current responsibilities better. * The ELManager is no longer optional in the code (it won't have a nil value under any circumstances). * The support for subscribing for headers was removed as it only worked with WebSockets and contributed significant complexity while bringing only a very minor advantage. * The `--web3-url` parameter has been deprecated in favor of a new `--el` parameter. The new parameter has a reasonable default value and supports specifying a different JWT for each connection. Each connection can also be configured with a different set of responsibilities (e.g. download deposits, validate blocks and/or produce blocks). On the command-line, these properties can be configured through URL properties stored in the #anchor part of the URL. In TOML files, they come with a very natural syntax (althrough the URL scheme is also supported). * The previously scattered EL-related state and logic is now moved to `eth1_monitor.nim` (this module will be renamed to `el_manager.nim` in a follow-up commit). State is assigned properly either to the `ELManager` or the to individual `ELConnection` objects where appropriate. The ELManager executes all Engine API requests against all attached EL nodes, in parallel. It compares their results and if there is a disagreement regarding the validity of a certain payload, this is detected and the beacon node is protected from publishing a block with a potential execution layer consensus bug in it. The BN provides metrics per EL node for the number of successful or failed requests for each type Engine API requests. If an EL node goes offline and connectivity is resoted later, we report the problem and the remedy in edge-triggered fashion. * More progress towards implementing Deneb block production in the VC and comparing the value of blocks produced by the EL and the builder API. * Adds a Makefile target for the zhejiang testnet
2023-03-05 01:40:21 +00:00
if bestPayloadIdx.isNone:
bestPayloadIdx = some idx
else:
if cmpGetPayloadResponses(req.read, requests[bestPayloadIdx.get].read) > 0:
bestPayloadIdx = some idx
if bestPayloadIdx.isSome:
return ok requests[bestPayloadIdx.get].read.asConsensusType
else:
return err()
proc waitELToSyncDeposits(connection: ELConnection,
minimalRequiredBlock: BlockHash) {.async.} =
var rpcClient = await connection.connectedRpcClient()
if connection.depositContractSyncStatus == DepositContractSyncStatus.synced:
return
var attempt = 0
while true:
try:
More metrics for the EL monitor (#4707) * `engine_api_response_time` provides a histogram for the Engine API response times for each unique pair ot URL and request type. * All engine API requests are now tracked Other changes: The client will no longer exit on start-up if it fails to connect to a properly configured EL node.
2023-03-09 17:29:04 +00:00
discard connection.trackedRequestWithTimeout(
"getBlockByHash",
rpcClient.getBlockByHash(minimalRequiredBlock),
Don't report an 'EL connection degraded' warning when the EL is not synced (#4761)
2023-03-21 22:19:36 +00:00
web3RequestsTimeout,
failureAllowed = true)
Support for driving multiple EL nodes from a single Nimbus BN (#4465) * Support for driving multiple EL nodes from a single Nimbus BN Full list of changes: * Eth1Monitor has been renamed to ELManager to match its current responsibilities better. * The ELManager is no longer optional in the code (it won't have a nil value under any circumstances). * The support for subscribing for headers was removed as it only worked with WebSockets and contributed significant complexity while bringing only a very minor advantage. * The `--web3-url` parameter has been deprecated in favor of a new `--el` parameter. The new parameter has a reasonable default value and supports specifying a different JWT for each connection. Each connection can also be configured with a different set of responsibilities (e.g. download deposits, validate blocks and/or produce blocks). On the command-line, these properties can be configured through URL properties stored in the #anchor part of the URL. In TOML files, they come with a very natural syntax (althrough the URL scheme is also supported). * The previously scattered EL-related state and logic is now moved to `eth1_monitor.nim` (this module will be renamed to `el_manager.nim` in a follow-up commit). State is assigned properly either to the `ELManager` or the to individual `ELConnection` objects where appropriate. The ELManager executes all Engine API requests against all attached EL nodes, in parallel. It compares their results and if there is a disagreement regarding the validity of a certain payload, this is detected and the beacon node is protected from publishing a block with a potential execution layer consensus bug in it. The BN provides metrics per EL node for the number of successful or failed requests for each type Engine API requests. If an EL node goes offline and connectivity is resoted later, we report the problem and the remedy in edge-triggered fashion. * More progress towards implementing Deneb block production in the VC and comparing the value of blocks produced by the EL and the builder API. * Adds a Makefile target for the zhejiang testnet
2023-03-05 01:40:21 +00:00
connection.depositContractSyncStatus = DepositContractSyncStatus.synced
return
except CancelledError as err:
trace "waitELToSyncDepositContract cancelled",
url = connection.engineUrl.url
raise err
except CatchableError as err:
connection.depositContractSyncStatus = DepositContractSyncStatus.notSynced
if attempt == 0:
warn "Failed to obtain the most recent known block from the execution " &
"layer node (the node is probably not synced)",
url = connection.engineUrl.url,
blk = minimalRequiredBlock,
err = err.msg
elif attempt mod 60 == 0:
# This warning will be produced every 30 minutes
warn "Still failing to obtain the most recent known block from the " &
"execution layer node (the node is probably still not synced)",
url = connection.engineUrl.url,
blk = minimalRequiredBlock,
err = err.msg
Prevent the build up of parallel 'waitELToSync' operations
2023-03-10 10:55:55 +00:00
inc attempt
Support for driving multiple EL nodes from a single Nimbus BN (#4465) * Support for driving multiple EL nodes from a single Nimbus BN Full list of changes: * Eth1Monitor has been renamed to ELManager to match its current responsibilities better. * The ELManager is no longer optional in the code (it won't have a nil value under any circumstances). * The support for subscribing for headers was removed as it only worked with WebSockets and contributed significant complexity while bringing only a very minor advantage. * The `--web3-url` parameter has been deprecated in favor of a new `--el` parameter. The new parameter has a reasonable default value and supports specifying a different JWT for each connection. Each connection can also be configured with a different set of responsibilities (e.g. download deposits, validate blocks and/or produce blocks). On the command-line, these properties can be configured through URL properties stored in the #anchor part of the URL. In TOML files, they come with a very natural syntax (althrough the URL scheme is also supported). * The previously scattered EL-related state and logic is now moved to `eth1_monitor.nim` (this module will be renamed to `el_manager.nim` in a follow-up commit). State is assigned properly either to the `ELManager` or the to individual `ELConnection` objects where appropriate. The ELManager executes all Engine API requests against all attached EL nodes, in parallel. It compares their results and if there is a disagreement regarding the validity of a certain payload, this is detected and the beacon node is protected from publishing a block with a potential execution layer consensus bug in it. The BN provides metrics per EL node for the number of successful or failed requests for each type Engine API requests. If an EL node goes offline and connectivity is resoted later, we report the problem and the remedy in edge-triggered fashion. * More progress towards implementing Deneb block production in the VC and comparing the value of blocks produced by the EL and the builder API. * Adds a Makefile target for the zhejiang testnet
2023-03-05 01:40:21 +00:00
await sleepAsync(seconds(30))
rpcClient = await connection.connectedRpcClient()
simplify ELMonitor fcU payload attributes handling (#4696)
2023-03-06 16:19:15 +00:00
func networkHasDepositContract(m: ELManager): bool =
Support for driving multiple EL nodes from a single Nimbus BN (#4465) * Support for driving multiple EL nodes from a single Nimbus BN Full list of changes: * Eth1Monitor has been renamed to ELManager to match its current responsibilities better. * The ELManager is no longer optional in the code (it won't have a nil value under any circumstances). * The support for subscribing for headers was removed as it only worked with WebSockets and contributed significant complexity while bringing only a very minor advantage. * The `--web3-url` parameter has been deprecated in favor of a new `--el` parameter. The new parameter has a reasonable default value and supports specifying a different JWT for each connection. Each connection can also be configured with a different set of responsibilities (e.g. download deposits, validate blocks and/or produce blocks). On the command-line, these properties can be configured through URL properties stored in the #anchor part of the URL. In TOML files, they come with a very natural syntax (althrough the URL scheme is also supported). * The previously scattered EL-related state and logic is now moved to `eth1_monitor.nim` (this module will be renamed to `el_manager.nim` in a follow-up commit). State is assigned properly either to the `ELManager` or the to individual `ELConnection` objects where appropriate. The ELManager executes all Engine API requests against all attached EL nodes, in parallel. It compares their results and if there is a disagreement regarding the validity of a certain payload, this is detected and the beacon node is protected from publishing a block with a potential execution layer consensus bug in it. The BN provides metrics per EL node for the number of successful or failed requests for each type Engine API requests. If an EL node goes offline and connectivity is resoted later, we report the problem and the remedy in edge-triggered fashion. * More progress towards implementing Deneb block production in the VC and comparing the value of blocks produced by the EL and the builder API. * Adds a Makefile target for the zhejiang testnet
2023-03-05 01:40:21 +00:00
not m.cfg.DEPOSIT_CONTRACT_ADDRESS.isDefaultValue
func mostRecentKnownBlock(m: ELManager): BlockHash =
if m.eth1Chain.finalizedDepositsMerkleizer.getChunkCount() > 0:
m.eth1Chain.finalizedBlockHash.asBlockHash
else:
m.depositContractBlockHash
proc selectConnectionForChainSyncing(m: ELManager): Future[ELConnection] {.async.} =
doAssert m.elConnections.len > 0
let connectionsFuts = mapIt(
m.elConnections,
if m.networkHasDepositContract:
FutureBase waitELToSyncDeposits(it, m.mostRecentKnownBlock)
else:
FutureBase connectedRpcClient(it))
# TODO: Ideally, the cancellation will be handled automatically
# by a helper like `firstCompletedFuture`
Prevent the build up of parallel 'waitELToSync' operations
2023-03-10 10:55:55 +00:00
let firstConnected = try:
await firstCompletedFuture(connectionsFuts)
except CancelledError as err:
for future in connectionsFuts:
BN+LC+SN: Fix cancellation deprecate warnings. (#5455) * Fix deprecation cancel() warnings in SN, BN, LC. * Fix missing import.
2023-09-22 11:06:27 +00:00
future.cancelSoon()
Prevent the build up of parallel 'waitELToSync' operations
2023-03-10 10:55:55 +00:00
raise err
Support for driving multiple EL nodes from a single Nimbus BN (#4465) * Support for driving multiple EL nodes from a single Nimbus BN Full list of changes: * Eth1Monitor has been renamed to ELManager to match its current responsibilities better. * The ELManager is no longer optional in the code (it won't have a nil value under any circumstances). * The support for subscribing for headers was removed as it only worked with WebSockets and contributed significant complexity while bringing only a very minor advantage. * The `--web3-url` parameter has been deprecated in favor of a new `--el` parameter. The new parameter has a reasonable default value and supports specifying a different JWT for each connection. Each connection can also be configured with a different set of responsibilities (e.g. download deposits, validate blocks and/or produce blocks). On the command-line, these properties can be configured through URL properties stored in the #anchor part of the URL. In TOML files, they come with a very natural syntax (althrough the URL scheme is also supported). * The previously scattered EL-related state and logic is now moved to `eth1_monitor.nim` (this module will be renamed to `el_manager.nim` in a follow-up commit). State is assigned properly either to the `ELManager` or the to individual `ELConnection` objects where appropriate. The ELManager executes all Engine API requests against all attached EL nodes, in parallel. It compares their results and if there is a disagreement regarding the validity of a certain payload, this is detected and the beacon node is protected from publishing a block with a potential execution layer consensus bug in it. The BN provides metrics per EL node for the number of successful or failed requests for each type Engine API requests. If an EL node goes offline and connectivity is resoted later, we report the problem and the remedy in edge-triggered fashion. * More progress towards implementing Deneb block production in the VC and comparing the value of blocks produced by the EL and the builder API. * Adds a Makefile target for the zhejiang testnet
2023-03-05 01:40:21 +00:00
for future in connectionsFuts:
if future != firstConnected:
BN+LC+SN: Fix cancellation deprecate warnings. (#5455) * Fix deprecation cancel() warnings in SN, BN, LC. * Fix missing import.
2023-09-22 11:06:27 +00:00
future.cancelSoon()
Support for driving multiple EL nodes from a single Nimbus BN (#4465) * Support for driving multiple EL nodes from a single Nimbus BN Full list of changes: * Eth1Monitor has been renamed to ELManager to match its current responsibilities better. * The ELManager is no longer optional in the code (it won't have a nil value under any circumstances). * The support for subscribing for headers was removed as it only worked with WebSockets and contributed significant complexity while bringing only a very minor advantage. * The `--web3-url` parameter has been deprecated in favor of a new `--el` parameter. The new parameter has a reasonable default value and supports specifying a different JWT for each connection. Each connection can also be configured with a different set of responsibilities (e.g. download deposits, validate blocks and/or produce blocks). On the command-line, these properties can be configured through URL properties stored in the #anchor part of the URL. In TOML files, they come with a very natural syntax (althrough the URL scheme is also supported). * The previously scattered EL-related state and logic is now moved to `eth1_monitor.nim` (this module will be renamed to `el_manager.nim` in a follow-up commit). State is assigned properly either to the `ELManager` or the to individual `ELConnection` objects where appropriate. The ELManager executes all Engine API requests against all attached EL nodes, in parallel. It compares their results and if there is a disagreement regarding the validity of a certain payload, this is detected and the beacon node is protected from publishing a block with a potential execution layer consensus bug in it. The BN provides metrics per EL node for the number of successful or failed requests for each type Engine API requests. If an EL node goes offline and connectivity is resoted later, we report the problem and the remedy in edge-triggered fashion. * More progress towards implementing Deneb block production in the VC and comparing the value of blocks produced by the EL and the builder API. * Adds a Makefile target for the zhejiang testnet
2023-03-05 01:40:21 +00:00
return m.elConnections[find(connectionsFuts, firstConnected)]
proc sendNewPayloadToSingleEL(connection: ELConnection,
payload: engine_api.ExecutionPayloadV1):
Future[PayloadStatusV1] {.async.} =
let rpcClient = await connection.connectedRpcClient()
return await rpcClient.engine_newPayloadV1(payload)
proc sendNewPayloadToSingleEL(connection: ELConnection,
payload: engine_api.ExecutionPayloadV2):
Future[PayloadStatusV1] {.async.} =
let rpcClient = await connection.connectedRpcClient()
return await rpcClient.engine_newPayloadV2(payload)
proc sendNewPayloadToSingleEL(connection: ELConnection,
https://github.com/ethereum/consensus-specs/pull/3359 (#5047)
2023-06-10 05:39:10 +00:00
payload: engine_api.ExecutionPayloadV3,
https://github.com/ethereum/consensus-specs/pull/3421 https://github.com/ethereum/execution-apis/pull/420 (#5147)
2023-06-30 08:14:20 +00:00
versioned_hashes: seq[engine_api.VersionedHash],
parent_beacon_block_root: FixedBytes[32]):
Support for driving multiple EL nodes from a single Nimbus BN (#4465) * Support for driving multiple EL nodes from a single Nimbus BN Full list of changes: * Eth1Monitor has been renamed to ELManager to match its current responsibilities better. * The ELManager is no longer optional in the code (it won't have a nil value under any circumstances). * The support for subscribing for headers was removed as it only worked with WebSockets and contributed significant complexity while bringing only a very minor advantage. * The `--web3-url` parameter has been deprecated in favor of a new `--el` parameter. The new parameter has a reasonable default value and supports specifying a different JWT for each connection. Each connection can also be configured with a different set of responsibilities (e.g. download deposits, validate blocks and/or produce blocks). On the command-line, these properties can be configured through URL properties stored in the #anchor part of the URL. In TOML files, they come with a very natural syntax (althrough the URL scheme is also supported). * The previously scattered EL-related state and logic is now moved to `eth1_monitor.nim` (this module will be renamed to `el_manager.nim` in a follow-up commit). State is assigned properly either to the `ELManager` or the to individual `ELConnection` objects where appropriate. The ELManager executes all Engine API requests against all attached EL nodes, in parallel. It compares their results and if there is a disagreement regarding the validity of a certain payload, this is detected and the beacon node is protected from publishing a block with a potential execution layer consensus bug in it. The BN provides metrics per EL node for the number of successful or failed requests for each type Engine API requests. If an EL node goes offline and connectivity is resoted later, we report the problem and the remedy in edge-triggered fashion. * More progress towards implementing Deneb block production in the VC and comparing the value of blocks produced by the EL and the builder API. * Adds a Makefile target for the zhejiang testnet
2023-03-05 01:40:21 +00:00
Future[PayloadStatusV1] {.async.} =
let rpcClient = await connection.connectedRpcClient()
https://github.com/ethereum/consensus-specs/pull/3421 https://github.com/ethereum/execution-apis/pull/420 (#5147)
2023-06-30 08:14:20 +00:00
return await rpcClient.engine_newPayloadV3(
payload, versioned_hashes, parent_beacon_block_root)
Bellatrix TTD detection (#3745) * Bellatrix TTD detection * Update beacon_chain/eth1/eth1_monitor.nim Co-authored-by: Etan Kissling <etan@status.im> * Update beacon_chain/nimbus_beacon_node.nim Co-authored-by: tersec <tersec@users.noreply.github.com> Co-authored-by: Etan Kissling <etan@status.im> Co-authored-by: tersec <tersec@users.noreply.github.com>
2022-06-15 02:38:27 +00:00
Require properly configured Engine API connection after the merge (#4006)
2022-08-22 19:44:40 +00:00
type
Support for driving multiple EL nodes from a single Nimbus BN (#4465) * Support for driving multiple EL nodes from a single Nimbus BN Full list of changes: * Eth1Monitor has been renamed to ELManager to match its current responsibilities better. * The ELManager is no longer optional in the code (it won't have a nil value under any circumstances). * The support for subscribing for headers was removed as it only worked with WebSockets and contributed significant complexity while bringing only a very minor advantage. * The `--web3-url` parameter has been deprecated in favor of a new `--el` parameter. The new parameter has a reasonable default value and supports specifying a different JWT for each connection. Each connection can also be configured with a different set of responsibilities (e.g. download deposits, validate blocks and/or produce blocks). On the command-line, these properties can be configured through URL properties stored in the #anchor part of the URL. In TOML files, they come with a very natural syntax (althrough the URL scheme is also supported). * The previously scattered EL-related state and logic is now moved to `eth1_monitor.nim` (this module will be renamed to `el_manager.nim` in a follow-up commit). State is assigned properly either to the `ELManager` or the to individual `ELConnection` objects where appropriate. The ELManager executes all Engine API requests against all attached EL nodes, in parallel. It compares their results and if there is a disagreement regarding the validity of a certain payload, this is detected and the beacon node is protected from publishing a block with a potential execution layer consensus bug in it. The BN provides metrics per EL node for the number of successful or failed requests for each type Engine API requests. If an EL node goes offline and connectivity is resoted later, we report the problem and the remedy in edge-triggered fashion. * More progress towards implementing Deneb block production in the VC and comparing the value of blocks produced by the EL and the builder API. * Adds a Makefile target for the zhejiang testnet
2023-03-05 01:40:21 +00:00
StatusRelation = enum
newStatusIsPreferable
oldStatusIsOk
disagreement
Bugfix: A single unsynced EL client may cause an inappropriate switch to optimistic mode.
2023-03-14 16:18:53 +00:00
func compareStatuses(newStatus, prevStatus: PayloadExecutionStatus): StatusRelation =
Support for driving multiple EL nodes from a single Nimbus BN (#4465) * Support for driving multiple EL nodes from a single Nimbus BN Full list of changes: * Eth1Monitor has been renamed to ELManager to match its current responsibilities better. * The ELManager is no longer optional in the code (it won't have a nil value under any circumstances). * The support for subscribing for headers was removed as it only worked with WebSockets and contributed significant complexity while bringing only a very minor advantage. * The `--web3-url` parameter has been deprecated in favor of a new `--el` parameter. The new parameter has a reasonable default value and supports specifying a different JWT for each connection. Each connection can also be configured with a different set of responsibilities (e.g. download deposits, validate blocks and/or produce blocks). On the command-line, these properties can be configured through URL properties stored in the #anchor part of the URL. In TOML files, they come with a very natural syntax (althrough the URL scheme is also supported). * The previously scattered EL-related state and logic is now moved to `eth1_monitor.nim` (this module will be renamed to `el_manager.nim` in a follow-up commit). State is assigned properly either to the `ELManager` or the to individual `ELConnection` objects where appropriate. The ELManager executes all Engine API requests against all attached EL nodes, in parallel. It compares their results and if there is a disagreement regarding the validity of a certain payload, this is detected and the beacon node is protected from publishing a block with a potential execution layer consensus bug in it. The BN provides metrics per EL node for the number of successful or failed requests for each type Engine API requests. If an EL node goes offline and connectivity is resoted later, we report the problem and the remedy in edge-triggered fashion. * More progress towards implementing Deneb block production in the VC and comparing the value of blocks produced by the EL and the builder API. * Adds a Makefile target for the zhejiang testnet
2023-03-05 01:40:21 +00:00
case prevStatus
of PayloadExecutionStatus.syncing:
if newStatus == PayloadExecutionStatus.syncing:
oldStatusIsOk
else:
newStatusIsPreferable
of PayloadExecutionStatus.valid:
case newStatus
of PayloadExecutionStatus.syncing,
PayloadExecutionStatus.accepted,
PayloadExecutionStatus.valid:
oldStatusIsOk
of PayloadExecutionStatus.invalid_block_hash,
PayloadExecutionStatus.invalid:
disagreement
of PayloadExecutionStatus.invalid:
case newStatus
of PayloadExecutionStatus.syncing,
PayloadExecutionStatus.invalid:
oldStatusIsOk
of PayloadExecutionStatus.valid,
PayloadExecutionStatus.accepted,
PayloadExecutionStatus.invalid_block_hash:
disagreement
of PayloadExecutionStatus.accepted:
case newStatus
of PayloadExecutionStatus.accepted,
PayloadExecutionStatus.syncing:
oldStatusIsOk
of PayloadExecutionStatus.valid:
newStatusIsPreferable
of PayloadExecutionStatus.invalid_block_hash,
PayloadExecutionStatus.invalid:
disagreement
of PayloadExecutionStatus.invalid_block_hash:
if newStatus == PayloadExecutionStatus.invalid_block_hash:
oldStatusIsOk
else:
disagreement
type
ELConsensusViolationDetector = object
selectedResponse: Option[int]
disagreementAlreadyDetected: bool
simplify ELMonitor fcU payload attributes handling (#4696)
2023-03-06 16:19:15 +00:00
func init(T: type ELConsensusViolationDetector): T =
Support for driving multiple EL nodes from a single Nimbus BN (#4465) * Support for driving multiple EL nodes from a single Nimbus BN Full list of changes: * Eth1Monitor has been renamed to ELManager to match its current responsibilities better. * The ELManager is no longer optional in the code (it won't have a nil value under any circumstances). * The support for subscribing for headers was removed as it only worked with WebSockets and contributed significant complexity while bringing only a very minor advantage. * The `--web3-url` parameter has been deprecated in favor of a new `--el` parameter. The new parameter has a reasonable default value and supports specifying a different JWT for each connection. Each connection can also be configured with a different set of responsibilities (e.g. download deposits, validate blocks and/or produce blocks). On the command-line, these properties can be configured through URL properties stored in the #anchor part of the URL. In TOML files, they come with a very natural syntax (althrough the URL scheme is also supported). * The previously scattered EL-related state and logic is now moved to `eth1_monitor.nim` (this module will be renamed to `el_manager.nim` in a follow-up commit). State is assigned properly either to the `ELManager` or the to individual `ELConnection` objects where appropriate. The ELManager executes all Engine API requests against all attached EL nodes, in parallel. It compares their results and if there is a disagreement regarding the validity of a certain payload, this is detected and the beacon node is protected from publishing a block with a potential execution layer consensus bug in it. The BN provides metrics per EL node for the number of successful or failed requests for each type Engine API requests. If an EL node goes offline and connectivity is resoted later, we report the problem and the remedy in edge-triggered fashion. * More progress towards implementing Deneb block production in the VC and comparing the value of blocks produced by the EL and the builder API. * Adds a Makefile target for the zhejiang testnet
2023-03-05 01:40:21 +00:00
ELConsensusViolationDetector(selectedResponse: none int,
disagreementAlreadyDetected: false)
proc processResponse[ELResponseType](
d: var ELConsensusViolationDetector,
connections: openArray[ELConnection],
requests: openArray[Future[ELResponseType]],
idx: int) =
Remove an overly agressive assertion (#4759)
2023-03-21 21:35:22 +00:00
if not requests[idx].completed:
return
Support for driving multiple EL nodes from a single Nimbus BN (#4465) * Support for driving multiple EL nodes from a single Nimbus BN Full list of changes: * Eth1Monitor has been renamed to ELManager to match its current responsibilities better. * The ELManager is no longer optional in the code (it won't have a nil value under any circumstances). * The support for subscribing for headers was removed as it only worked with WebSockets and contributed significant complexity while bringing only a very minor advantage. * The `--web3-url` parameter has been deprecated in favor of a new `--el` parameter. The new parameter has a reasonable default value and supports specifying a different JWT for each connection. Each connection can also be configured with a different set of responsibilities (e.g. download deposits, validate blocks and/or produce blocks). On the command-line, these properties can be configured through URL properties stored in the #anchor part of the URL. In TOML files, they come with a very natural syntax (althrough the URL scheme is also supported). * The previously scattered EL-related state and logic is now moved to `eth1_monitor.nim` (this module will be renamed to `el_manager.nim` in a follow-up commit). State is assigned properly either to the `ELManager` or the to individual `ELConnection` objects where appropriate. The ELManager executes all Engine API requests against all attached EL nodes, in parallel. It compares their results and if there is a disagreement regarding the validity of a certain payload, this is detected and the beacon node is protected from publishing a block with a potential execution layer consensus bug in it. The BN provides metrics per EL node for the number of successful or failed requests for each type Engine API requests. If an EL node goes offline and connectivity is resoted later, we report the problem and the remedy in edge-triggered fashion. * More progress towards implementing Deneb block production in the VC and comparing the value of blocks produced by the EL and the builder API. * Adds a Makefile target for the zhejiang testnet
2023-03-05 01:40:21 +00:00
let status = try: requests[idx].read.status
except CatchableError: raiseAssert "checked above"
if d.selectedResponse.isNone:
d.selectedResponse = some idx
elif not d.disagreementAlreadyDetected:
let prevStatus = try: requests[d.selectedResponse.get].read.status
except CatchableError: raiseAssert "previously checked"
case compareStatuses(status, prevStatus)
of newStatusIsPreferable:
d.selectedResponse = some idx
of oldStatusIsOk:
discard
of disagreement:
d.disagreementAlreadyDetected = true
error "Execution layer consensus violation detected",
responseType = name(ELResponseType),
url1 = connections[d.selectedResponse.get].engineUrl.url,
status1 = prevStatus,
url2 = connections[idx].engineUrl.url,
status2 = status
https://github.com/ethereum/consensus-specs/pull/3421 https://github.com/ethereum/execution-apis/pull/420 (#5147)
2023-06-30 08:14:20 +00:00
proc sendNewPayload*(m: ELManager, blck: SomeForkyBeaconBlock):
Support for driving multiple EL nodes from a single Nimbus BN (#4465) * Support for driving multiple EL nodes from a single Nimbus BN Full list of changes: * Eth1Monitor has been renamed to ELManager to match its current responsibilities better. * The ELManager is no longer optional in the code (it won't have a nil value under any circumstances). * The support for subscribing for headers was removed as it only worked with WebSockets and contributed significant complexity while bringing only a very minor advantage. * The `--web3-url` parameter has been deprecated in favor of a new `--el` parameter. The new parameter has a reasonable default value and supports specifying a different JWT for each connection. Each connection can also be configured with a different set of responsibilities (e.g. download deposits, validate blocks and/or produce blocks). On the command-line, these properties can be configured through URL properties stored in the #anchor part of the URL. In TOML files, they come with a very natural syntax (althrough the URL scheme is also supported). * The previously scattered EL-related state and logic is now moved to `eth1_monitor.nim` (this module will be renamed to `el_manager.nim` in a follow-up commit). State is assigned properly either to the `ELManager` or the to individual `ELConnection` objects where appropriate. The ELManager executes all Engine API requests against all attached EL nodes, in parallel. It compares their results and if there is a disagreement regarding the validity of a certain payload, this is detected and the beacon node is protected from publishing a block with a potential execution layer consensus bug in it. The BN provides metrics per EL node for the number of successful or failed requests for each type Engine API requests. If an EL node goes offline and connectivity is resoted later, we report the problem and the remedy in edge-triggered fashion. * More progress towards implementing Deneb block production in the VC and comparing the value of blocks produced by the EL and the builder API. * Adds a Makefile target for the zhejiang testnet
2023-03-05 01:40:21 +00:00
Future[PayloadExecutionStatus] {.async.} =
let
earlyDeadline = sleepAsync(chronos.seconds 1)
More metrics for the EL monitor (#4707) * `engine_api_response_time` provides a histogram for the Engine API response times for each unique pair ot URL and request type. * All engine API requests are now tracked Other changes: The client will no longer exit on start-up if it fails to connect to a properly configured EL node.
2023-03-09 17:29:04 +00:00
startTime = Moment.now
Support for driving multiple EL nodes from a single Nimbus BN (#4465) * Support for driving multiple EL nodes from a single Nimbus BN Full list of changes: * Eth1Monitor has been renamed to ELManager to match its current responsibilities better. * The ELManager is no longer optional in the code (it won't have a nil value under any circumstances). * The support for subscribing for headers was removed as it only worked with WebSockets and contributed significant complexity while bringing only a very minor advantage. * The `--web3-url` parameter has been deprecated in favor of a new `--el` parameter. The new parameter has a reasonable default value and supports specifying a different JWT for each connection. Each connection can also be configured with a different set of responsibilities (e.g. download deposits, validate blocks and/or produce blocks). On the command-line, these properties can be configured through URL properties stored in the #anchor part of the URL. In TOML files, they come with a very natural syntax (althrough the URL scheme is also supported). * The previously scattered EL-related state and logic is now moved to `eth1_monitor.nim` (this module will be renamed to `el_manager.nim` in a follow-up commit). State is assigned properly either to the `ELManager` or the to individual `ELConnection` objects where appropriate. The ELManager executes all Engine API requests against all attached EL nodes, in parallel. It compares their results and if there is a disagreement regarding the validity of a certain payload, this is detected and the beacon node is protected from publishing a block with a potential execution layer consensus bug in it. The BN provides metrics per EL node for the number of successful or failed requests for each type Engine API requests. If an EL node goes offline and connectivity is resoted later, we report the problem and the remedy in edge-triggered fashion. * More progress towards implementing Deneb block production in the VC and comparing the value of blocks produced by the EL and the builder API. * Adds a Makefile target for the zhejiang testnet
2023-03-05 01:40:21 +00:00
deadline = sleepAsync(NEWPAYLOAD_TIMEOUT)
https://github.com/ethereum/consensus-specs/pull/3421 https://github.com/ethereum/execution-apis/pull/420 (#5147)
2023-06-30 08:14:20 +00:00
payload = blck.body.execution_payload.asEngineExecutionPayload
Support for driving multiple EL nodes from a single Nimbus BN (#4465) * Support for driving multiple EL nodes from a single Nimbus BN Full list of changes: * Eth1Monitor has been renamed to ELManager to match its current responsibilities better. * The ELManager is no longer optional in the code (it won't have a nil value under any circumstances). * The support for subscribing for headers was removed as it only worked with WebSockets and contributed significant complexity while bringing only a very minor advantage. * The `--web3-url` parameter has been deprecated in favor of a new `--el` parameter. The new parameter has a reasonable default value and supports specifying a different JWT for each connection. Each connection can also be configured with a different set of responsibilities (e.g. download deposits, validate blocks and/or produce blocks). On the command-line, these properties can be configured through URL properties stored in the #anchor part of the URL. In TOML files, they come with a very natural syntax (althrough the URL scheme is also supported). * The previously scattered EL-related state and logic is now moved to `eth1_monitor.nim` (this module will be renamed to `el_manager.nim` in a follow-up commit). State is assigned properly either to the `ELManager` or the to individual `ELConnection` objects where appropriate. The ELManager executes all Engine API requests against all attached EL nodes, in parallel. It compares their results and if there is a disagreement regarding the validity of a certain payload, this is detected and the beacon node is protected from publishing a block with a potential execution layer consensus bug in it. The BN provides metrics per EL node for the number of successful or failed requests for each type Engine API requests. If an EL node goes offline and connectivity is resoted later, we report the problem and the remedy in edge-triggered fashion. * More progress towards implementing Deneb block production in the VC and comparing the value of blocks produced by the EL and the builder API. * Adds a Makefile target for the zhejiang testnet
2023-03-05 01:40:21 +00:00
requests = m.elConnections.mapIt:
https://github.com/ethereum/consensus-specs/pull/3359 (#5047)
2023-06-10 05:39:10 +00:00
let req =
when payload is engine_api.ExecutionPayloadV3:
# https://github.com/ethereum/consensus-specs/blob/v1.4.0-alpha.1/specs/deneb/beacon-chain.md#process_execution_payload
# Verify the execution payload is valid
# [Modified in Deneb] Pass `versioned_hashes` to Execution Engine
let versioned_hashes = mapIt(
https://github.com/ethereum/consensus-specs/pull/3421 https://github.com/ethereum/execution-apis/pull/420 (#5147)
2023-06-30 08:14:20 +00:00
blck.body.blob_kzg_commitments,
https://github.com/ethereum/consensus-specs/pull/3359 (#5047)
2023-06-10 05:39:10 +00:00
engine_api.VersionedHash(kzg_commitment_to_versioned_hash(it)))
https://github.com/ethereum/consensus-specs/pull/3421 https://github.com/ethereum/execution-apis/pull/420 (#5147)
2023-06-30 08:14:20 +00:00
sendNewPayloadToSingleEL(
it, payload, versioned_hashes,
FixedBytes[32] blck.parent_root.data)
https://github.com/ethereum/consensus-specs/pull/3359 (#5047)
2023-06-10 05:39:10 +00:00
elif payload is engine_api.ExecutionPayloadV1 or
payload is engine_api.ExecutionPayloadV2:
sendNewPayloadToSingleEL(it, payload)
else:
static: doAssert false
More metrics for the EL monitor (#4707) * `engine_api_response_time` provides a histogram for the Engine API response times for each unique pair ot URL and request type. * All engine API requests are now tracked Other changes: The client will no longer exit on start-up if it fails to connect to a properly configured EL node.
2023-03-09 17:29:04 +00:00
trackEngineApiRequest(it, req, "newPayload", startTime, deadline)
Support for driving multiple EL nodes from a single Nimbus BN (#4465) * Support for driving multiple EL nodes from a single Nimbus BN Full list of changes: * Eth1Monitor has been renamed to ELManager to match its current responsibilities better. * The ELManager is no longer optional in the code (it won't have a nil value under any circumstances). * The support for subscribing for headers was removed as it only worked with WebSockets and contributed significant complexity while bringing only a very minor advantage. * The `--web3-url` parameter has been deprecated in favor of a new `--el` parameter. The new parameter has a reasonable default value and supports specifying a different JWT for each connection. Each connection can also be configured with a different set of responsibilities (e.g. download deposits, validate blocks and/or produce blocks). On the command-line, these properties can be configured through URL properties stored in the #anchor part of the URL. In TOML files, they come with a very natural syntax (althrough the URL scheme is also supported). * The previously scattered EL-related state and logic is now moved to `eth1_monitor.nim` (this module will be renamed to `el_manager.nim` in a follow-up commit). State is assigned properly either to the `ELManager` or the to individual `ELConnection` objects where appropriate. The ELManager executes all Engine API requests against all attached EL nodes, in parallel. It compares their results and if there is a disagreement regarding the validity of a certain payload, this is detected and the beacon node is protected from publishing a block with a potential execution layer consensus bug in it. The BN provides metrics per EL node for the number of successful or failed requests for each type Engine API requests. If an EL node goes offline and connectivity is resoted later, we report the problem and the remedy in edge-triggered fashion. * More progress towards implementing Deneb block production in the VC and comparing the value of blocks produced by the EL and the builder API. * Adds a Makefile target for the zhejiang testnet
2023-03-05 01:40:21 +00:00
req
requestsCompleted = allFutures(requests)
await requestsCompleted or earlyDeadline
Require properly configured Engine API connection after the merge (#4006)
2022-08-22 19:44:40 +00:00
Support for driving multiple EL nodes from a single Nimbus BN (#4465) * Support for driving multiple EL nodes from a single Nimbus BN Full list of changes: * Eth1Monitor has been renamed to ELManager to match its current responsibilities better. * The ELManager is no longer optional in the code (it won't have a nil value under any circumstances). * The support for subscribing for headers was removed as it only worked with WebSockets and contributed significant complexity while bringing only a very minor advantage. * The `--web3-url` parameter has been deprecated in favor of a new `--el` parameter. The new parameter has a reasonable default value and supports specifying a different JWT for each connection. Each connection can also be configured with a different set of responsibilities (e.g. download deposits, validate blocks and/or produce blocks). On the command-line, these properties can be configured through URL properties stored in the #anchor part of the URL. In TOML files, they come with a very natural syntax (althrough the URL scheme is also supported). * The previously scattered EL-related state and logic is now moved to `eth1_monitor.nim` (this module will be renamed to `el_manager.nim` in a follow-up commit). State is assigned properly either to the `ELManager` or the to individual `ELConnection` objects where appropriate. The ELManager executes all Engine API requests against all attached EL nodes, in parallel. It compares their results and if there is a disagreement regarding the validity of a certain payload, this is detected and the beacon node is protected from publishing a block with a potential execution layer consensus bug in it. The BN provides metrics per EL node for the number of successful or failed requests for each type Engine API requests. If an EL node goes offline and connectivity is resoted later, we report the problem and the remedy in edge-triggered fashion. * More progress towards implementing Deneb block production in the VC and comparing the value of blocks produced by the EL and the builder API. * Adds a Makefile target for the zhejiang testnet
2023-03-05 01:40:21 +00:00
var
stillPending = newSeq[Future[PayloadStatusV1]]()
responseProcessor = init ELConsensusViolationDetector
for idx, req in requests:
if not req.finished:
stillPending.add req
Remove an overly agressive assertion (#4759)
2023-03-21 21:35:22 +00:00
elif req.completed:
Support for driving multiple EL nodes from a single Nimbus BN (#4465) * Support for driving multiple EL nodes from a single Nimbus BN Full list of changes: * Eth1Monitor has been renamed to ELManager to match its current responsibilities better. * The ELManager is no longer optional in the code (it won't have a nil value under any circumstances). * The support for subscribing for headers was removed as it only worked with WebSockets and contributed significant complexity while bringing only a very minor advantage. * The `--web3-url` parameter has been deprecated in favor of a new `--el` parameter. The new parameter has a reasonable default value and supports specifying a different JWT for each connection. Each connection can also be configured with a different set of responsibilities (e.g. download deposits, validate blocks and/or produce blocks). On the command-line, these properties can be configured through URL properties stored in the #anchor part of the URL. In TOML files, they come with a very natural syntax (althrough the URL scheme is also supported). * The previously scattered EL-related state and logic is now moved to `eth1_monitor.nim` (this module will be renamed to `el_manager.nim` in a follow-up commit). State is assigned properly either to the `ELManager` or the to individual `ELConnection` objects where appropriate. The ELManager executes all Engine API requests against all attached EL nodes, in parallel. It compares their results and if there is a disagreement regarding the validity of a certain payload, this is detected and the beacon node is protected from publishing a block with a potential execution layer consensus bug in it. The BN provides metrics per EL node for the number of successful or failed requests for each type Engine API requests. If an EL node goes offline and connectivity is resoted later, we report the problem and the remedy in edge-triggered fashion. * More progress towards implementing Deneb block production in the VC and comparing the value of blocks produced by the EL and the builder API. * Adds a Makefile target for the zhejiang testnet
2023-03-05 01:40:21 +00:00
responseProcessor.processResponse(m.elConnections, requests, idx)
if responseProcessor.disagreementAlreadyDetected:
return PayloadExecutionStatus.invalid
elif responseProcessor.selectedResponse.isSome:
return requests[responseProcessor.selectedResponse.get].read.status
await requestsCompleted or deadline
for idx, req in requests:
if req.completed and req in stillPending:
responseProcessor.processResponse(m.elConnections, requests, idx)
return if responseProcessor.disagreementAlreadyDetected:
PayloadExecutionStatus.invalid
elif responseProcessor.selectedResponse.isSome:
requests[responseProcessor.selectedResponse.get].read.status
else:
PayloadExecutionStatus.syncing
proc forkchoiceUpdatedForSingleEL(
connection: ELConnection,
state: ref ForkchoiceStateV1,
always use fcUV2 in shapella even for non-proposer fcUs (#4817) * always use fcUV2 in shapella even for non-proposer fcUs * avoid template/proc naming conflict with libp2p/signed_envelope.nim having a payload proc
2023-04-17 14:17:52 +00:00
payloadAttributes: Option[PayloadAttributesV1] |
use fcU V3 in Deneb/Cancun (#5293)
2023-08-15 23:00:35 +00:00
Option[PayloadAttributesV2] |
Option[PayloadAttributesV3]):
Support for driving multiple EL nodes from a single Nimbus BN (#4465) * Support for driving multiple EL nodes from a single Nimbus BN Full list of changes: * Eth1Monitor has been renamed to ELManager to match its current responsibilities better. * The ELManager is no longer optional in the code (it won't have a nil value under any circumstances). * The support for subscribing for headers was removed as it only worked with WebSockets and contributed significant complexity while bringing only a very minor advantage. * The `--web3-url` parameter has been deprecated in favor of a new `--el` parameter. The new parameter has a reasonable default value and supports specifying a different JWT for each connection. Each connection can also be configured with a different set of responsibilities (e.g. download deposits, validate blocks and/or produce blocks). On the command-line, these properties can be configured through URL properties stored in the #anchor part of the URL. In TOML files, they come with a very natural syntax (althrough the URL scheme is also supported). * The previously scattered EL-related state and logic is now moved to `eth1_monitor.nim` (this module will be renamed to `el_manager.nim` in a follow-up commit). State is assigned properly either to the `ELManager` or the to individual `ELConnection` objects where appropriate. The ELManager executes all Engine API requests against all attached EL nodes, in parallel. It compares their results and if there is a disagreement regarding the validity of a certain payload, this is detected and the beacon node is protected from publishing a block with a potential execution layer consensus bug in it. The BN provides metrics per EL node for the number of successful or failed requests for each type Engine API requests. If an EL node goes offline and connectivity is resoted later, we report the problem and the remedy in edge-triggered fashion. * More progress towards implementing Deneb block production in the VC and comparing the value of blocks produced by the EL and the builder API. * Adds a Makefile target for the zhejiang testnet
2023-03-05 01:40:21 +00:00
Future[PayloadStatusV1] {.async.} =
let
rpcClient = await connection.connectedRpcClient()
response = await rpcClient.forkchoiceUpdated(state[], payloadAttributes)
if response.payloadStatus.status notin {syncing, valid, invalid}:
debug "Invalid fork-choice updated response from the EL",
payloadStatus = response.payloadStatus
return
if response.payloadStatus.status == PayloadExecutionStatus.valid and
response.payloadId.isSome:
connection.lastPayloadId = response.payloadId
return response.payloadStatus
proc forkchoiceUpdated*(m: ELManager,
always use fcUV2 in shapella even for non-proposer fcUs (#4817) * always use fcUV2 in shapella even for non-proposer fcUs * avoid template/proc naming conflict with libp2p/signed_envelope.nim having a payload proc
2023-04-17 14:17:52 +00:00
headBlockHash, safeBlockHash,
finalizedBlockHash: Eth2Digest,
payloadAttributes: Option[PayloadAttributesV1] |
use fcU V3 in Deneb/Cancun (#5293)
2023-08-15 23:00:35 +00:00
Option[PayloadAttributesV2] |
Option[PayloadAttributesV3]):
Support for driving multiple EL nodes from a single Nimbus BN (#4465) * Support for driving multiple EL nodes from a single Nimbus BN Full list of changes: * Eth1Monitor has been renamed to ELManager to match its current responsibilities better. * The ELManager is no longer optional in the code (it won't have a nil value under any circumstances). * The support for subscribing for headers was removed as it only worked with WebSockets and contributed significant complexity while bringing only a very minor advantage. * The `--web3-url` parameter has been deprecated in favor of a new `--el` parameter. The new parameter has a reasonable default value and supports specifying a different JWT for each connection. Each connection can also be configured with a different set of responsibilities (e.g. download deposits, validate blocks and/or produce blocks). On the command-line, these properties can be configured through URL properties stored in the #anchor part of the URL. In TOML files, they come with a very natural syntax (althrough the URL scheme is also supported). * The previously scattered EL-related state and logic is now moved to `eth1_monitor.nim` (this module will be renamed to `el_manager.nim` in a follow-up commit). State is assigned properly either to the `ELManager` or the to individual `ELConnection` objects where appropriate. The ELManager executes all Engine API requests against all attached EL nodes, in parallel. It compares their results and if there is a disagreement regarding the validity of a certain payload, this is detected and the beacon node is protected from publishing a block with a potential execution layer consensus bug in it. The BN provides metrics per EL node for the number of successful or failed requests for each type Engine API requests. If an EL node goes offline and connectivity is resoted later, we report the problem and the remedy in edge-triggered fashion. * More progress towards implementing Deneb block production in the VC and comparing the value of blocks produced by the EL and the builder API. * Adds a Makefile target for the zhejiang testnet
2023-03-05 01:40:21 +00:00
Future[(PayloadExecutionStatus, Option[BlockHash])] {.async.} =
doAssert not headBlockHash.isZero
# Allow finalizedBlockHash to be 0 to avoid sync deadlocks.
#
# https://github.com/ethereum/EIPs/blob/master/EIPS/eip-3675.md#pos-events
# has "Before the first finalized block occurs in the system the finalized
# block hash provided by this event is stubbed with
# `0x0000000000000000000000000000000000000000000000000000000000000000`."
# and
more consensus-specs v1.3.0 bumps (#4898) Updates more consensus-specs references to v1.3.0. This PR only includes updates for sections otherwise unmodified.
2023-05-05 20:38:51 +00:00
# https://github.com/ethereum/consensus-specs/blob/v1.3.0/specs/bellatrix/validator.md#executionpayload
Support for driving multiple EL nodes from a single Nimbus BN (#4465) * Support for driving multiple EL nodes from a single Nimbus BN Full list of changes: * Eth1Monitor has been renamed to ELManager to match its current responsibilities better. * The ELManager is no longer optional in the code (it won't have a nil value under any circumstances). * The support for subscribing for headers was removed as it only worked with WebSockets and contributed significant complexity while bringing only a very minor advantage. * The `--web3-url` parameter has been deprecated in favor of a new `--el` parameter. The new parameter has a reasonable default value and supports specifying a different JWT for each connection. Each connection can also be configured with a different set of responsibilities (e.g. download deposits, validate blocks and/or produce blocks). On the command-line, these properties can be configured through URL properties stored in the #anchor part of the URL. In TOML files, they come with a very natural syntax (althrough the URL scheme is also supported). * The previously scattered EL-related state and logic is now moved to `eth1_monitor.nim` (this module will be renamed to `el_manager.nim` in a follow-up commit). State is assigned properly either to the `ELManager` or the to individual `ELConnection` objects where appropriate. The ELManager executes all Engine API requests against all attached EL nodes, in parallel. It compares their results and if there is a disagreement regarding the validity of a certain payload, this is detected and the beacon node is protected from publishing a block with a potential execution layer consensus bug in it. The BN provides metrics per EL node for the number of successful or failed requests for each type Engine API requests. If an EL node goes offline and connectivity is resoted later, we report the problem and the remedy in edge-triggered fashion. * More progress towards implementing Deneb block production in the VC and comparing the value of blocks produced by the EL and the builder API. * Adds a Makefile target for the zhejiang testnet
2023-03-05 01:40:21 +00:00
# notes "`finalized_block_hash` is the hash of the latest finalized execution
# payload (`Hash32()` if none yet finalized)"
if m.elConnections.len == 0:
return (PayloadExecutionStatus.syncing, none BlockHash)
use fcU V3 in Deneb/Cancun (#5293)
2023-08-15 23:00:35 +00:00
when payloadAttributes is Option[PayloadAttributesV3]:
template payloadAttributesV3(): auto =
always use fcUV2 in shapella even for non-proposer fcUs (#4817) * always use fcUV2 in shapella even for non-proposer fcUs * avoid template/proc naming conflict with libp2p/signed_envelope.nim having a payload proc
2023-04-17 14:17:52 +00:00
if payloadAttributes.isSome:
payloadAttributes.get
else:
# As timestamp and prevRandao are both 0, won't false-positive match
use fcU V3 in Deneb/Cancun (#5293)
2023-08-15 23:00:35 +00:00
(static(default(PayloadAttributesV3)))
elif payloadAttributes is Option[PayloadAttributesV2]:
template payloadAttributesV3(): auto =
if payloadAttributes.isSome:
PayloadAttributesV3(
timestamp: payloadAttributes.get.timestamp,
prevRandao: payloadAttributes.get.prevRandao,
suggestedFeeRecipient: payloadAttributes.get.suggestedFeeRecipient,
withdrawals: payloadAttributes.get.withdrawals,
parentBeaconBlockRoot: static(default(FixedBytes[32])))
else:
# As timestamp and prevRandao are both 0, won't false-positive match
(static(default(PayloadAttributesV3)))
always use fcUV2 in shapella even for non-proposer fcUs (#4817) * always use fcUV2 in shapella even for non-proposer fcUs * avoid template/proc naming conflict with libp2p/signed_envelope.nim having a payload proc
2023-04-17 14:17:52 +00:00
elif payloadAttributes is Option[PayloadAttributesV1]:
use fcU V3 in Deneb/Cancun (#5293)
2023-08-15 23:00:35 +00:00
template payloadAttributesV3(): auto =
always use fcUV2 in shapella even for non-proposer fcUs (#4817) * always use fcUV2 in shapella even for non-proposer fcUs * avoid template/proc naming conflict with libp2p/signed_envelope.nim having a payload proc
2023-04-17 14:17:52 +00:00
if payloadAttributes.isSome:
use fcU V3 in Deneb/Cancun (#5293)
2023-08-15 23:00:35 +00:00
PayloadAttributesV3(
always use fcUV2 in shapella even for non-proposer fcUs (#4817) * always use fcUV2 in shapella even for non-proposer fcUs * avoid template/proc naming conflict with libp2p/signed_envelope.nim having a payload proc
2023-04-17 14:17:52 +00:00
timestamp: payloadAttributes.get.timestamp,
prevRandao: payloadAttributes.get.prevRandao,
suggestedFeeRecipient: payloadAttributes.get.suggestedFeeRecipient,
use fcU V3 in Deneb/Cancun (#5293)
2023-08-15 23:00:35 +00:00
withdrawals: @[],
parentBeaconBlockRoot: static(default(FixedBytes[32])))
always use fcUV2 in shapella even for non-proposer fcUs (#4817) * always use fcUV2 in shapella even for non-proposer fcUs * avoid template/proc naming conflict with libp2p/signed_envelope.nim having a payload proc
2023-04-17 14:17:52 +00:00
else:
# As timestamp and prevRandao are both 0, won't false-positive match
use fcU V3 in Deneb/Cancun (#5293)
2023-08-15 23:00:35 +00:00
(static(default(PayloadAttributesV3)))
simplify ELMonitor fcU payload attributes handling (#4696)
2023-03-06 16:19:15 +00:00
else:
static: doAssert false
Support for driving multiple EL nodes from a single Nimbus BN (#4465) * Support for driving multiple EL nodes from a single Nimbus BN Full list of changes: * Eth1Monitor has been renamed to ELManager to match its current responsibilities better. * The ELManager is no longer optional in the code (it won't have a nil value under any circumstances). * The support for subscribing for headers was removed as it only worked with WebSockets and contributed significant complexity while bringing only a very minor advantage. * The `--web3-url` parameter has been deprecated in favor of a new `--el` parameter. The new parameter has a reasonable default value and supports specifying a different JWT for each connection. Each connection can also be configured with a different set of responsibilities (e.g. download deposits, validate blocks and/or produce blocks). On the command-line, these properties can be configured through URL properties stored in the #anchor part of the URL. In TOML files, they come with a very natural syntax (althrough the URL scheme is also supported). * The previously scattered EL-related state and logic is now moved to `eth1_monitor.nim` (this module will be renamed to `el_manager.nim` in a follow-up commit). State is assigned properly either to the `ELManager` or the to individual `ELConnection` objects where appropriate. The ELManager executes all Engine API requests against all attached EL nodes, in parallel. It compares their results and if there is a disagreement regarding the validity of a certain payload, this is detected and the beacon node is protected from publishing a block with a potential execution layer consensus bug in it. The BN provides metrics per EL node for the number of successful or failed requests for each type Engine API requests. If an EL node goes offline and connectivity is resoted later, we report the problem and the remedy in edge-triggered fashion. * More progress towards implementing Deneb block production in the VC and comparing the value of blocks produced by the EL and the builder API. * Adds a Makefile target for the zhejiang testnet
2023-03-05 01:40:21 +00:00
let
state = newClone ForkchoiceStateV1(
headBlockHash: headBlockHash.asBlockHash,
safeBlockHash: safeBlockHash.asBlockHash,
finalizedBlockHash: finalizedBlockHash.asBlockHash)
earlyDeadline = sleepAsync(chronos.seconds 1)
More metrics for the EL monitor (#4707) * `engine_api_response_time` provides a histogram for the Engine API response times for each unique pair ot URL and request type. * All engine API requests are now tracked Other changes: The client will no longer exit on start-up if it fails to connect to a properly configured EL node.
2023-03-09 17:29:04 +00:00
startTime = Moment.now
Support for driving multiple EL nodes from a single Nimbus BN (#4465) * Support for driving multiple EL nodes from a single Nimbus BN Full list of changes: * Eth1Monitor has been renamed to ELManager to match its current responsibilities better. * The ELManager is no longer optional in the code (it won't have a nil value under any circumstances). * The support for subscribing for headers was removed as it only worked with WebSockets and contributed significant complexity while bringing only a very minor advantage. * The `--web3-url` parameter has been deprecated in favor of a new `--el` parameter. The new parameter has a reasonable default value and supports specifying a different JWT for each connection. Each connection can also be configured with a different set of responsibilities (e.g. download deposits, validate blocks and/or produce blocks). On the command-line, these properties can be configured through URL properties stored in the #anchor part of the URL. In TOML files, they come with a very natural syntax (althrough the URL scheme is also supported). * The previously scattered EL-related state and logic is now moved to `eth1_monitor.nim` (this module will be renamed to `el_manager.nim` in a follow-up commit). State is assigned properly either to the `ELManager` or the to individual `ELConnection` objects where appropriate. The ELManager executes all Engine API requests against all attached EL nodes, in parallel. It compares their results and if there is a disagreement regarding the validity of a certain payload, this is detected and the beacon node is protected from publishing a block with a potential execution layer consensus bug in it. The BN provides metrics per EL node for the number of successful or failed requests for each type Engine API requests. If an EL node goes offline and connectivity is resoted later, we report the problem and the remedy in edge-triggered fashion. * More progress towards implementing Deneb block production in the VC and comparing the value of blocks produced by the EL and the builder API. * Adds a Makefile target for the zhejiang testnet
2023-03-05 01:40:21 +00:00
deadline = sleepAsync(FORKCHOICEUPDATED_TIMEOUT)
requests = m.elConnections.mapIt:
let req = it.forkchoiceUpdatedForSingleEL(state, payloadAttributes)
More metrics for the EL monitor (#4707) * `engine_api_response_time` provides a histogram for the Engine API response times for each unique pair ot URL and request type. * All engine API requests are now tracked Other changes: The client will no longer exit on start-up if it fails to connect to a properly configured EL node.
2023-03-09 17:29:04 +00:00
trackEngineApiRequest(it, req, "forkchoiceUpdated", startTime, deadline)
Support for driving multiple EL nodes from a single Nimbus BN (#4465) * Support for driving multiple EL nodes from a single Nimbus BN Full list of changes: * Eth1Monitor has been renamed to ELManager to match its current responsibilities better. * The ELManager is no longer optional in the code (it won't have a nil value under any circumstances). * The support for subscribing for headers was removed as it only worked with WebSockets and contributed significant complexity while bringing only a very minor advantage. * The `--web3-url` parameter has been deprecated in favor of a new `--el` parameter. The new parameter has a reasonable default value and supports specifying a different JWT for each connection. Each connection can also be configured with a different set of responsibilities (e.g. download deposits, validate blocks and/or produce blocks). On the command-line, these properties can be configured through URL properties stored in the #anchor part of the URL. In TOML files, they come with a very natural syntax (althrough the URL scheme is also supported). * The previously scattered EL-related state and logic is now moved to `eth1_monitor.nim` (this module will be renamed to `el_manager.nim` in a follow-up commit). State is assigned properly either to the `ELManager` or the to individual `ELConnection` objects where appropriate. The ELManager executes all Engine API requests against all attached EL nodes, in parallel. It compares their results and if there is a disagreement regarding the validity of a certain payload, this is detected and the beacon node is protected from publishing a block with a potential execution layer consensus bug in it. The BN provides metrics per EL node for the number of successful or failed requests for each type Engine API requests. If an EL node goes offline and connectivity is resoted later, we report the problem and the remedy in edge-triggered fashion. * More progress towards implementing Deneb block production in the VC and comparing the value of blocks produced by the EL and the builder API. * Adds a Makefile target for the zhejiang testnet
2023-03-05 01:40:21 +00:00
req
requestsCompleted = allFutures(requests)
await requestsCompleted or earlyDeadline
var
stillPending = newSeq[Future[PayloadStatusV1]]()
responseProcessor = init ELConsensusViolationDetector
for idx, req in requests:
if not req.finished:
stillPending.add req
Remove an overly agressive assertion (#4759)
2023-03-21 21:35:22 +00:00
elif req.completed:
Support for driving multiple EL nodes from a single Nimbus BN (#4465) * Support for driving multiple EL nodes from a single Nimbus BN Full list of changes: * Eth1Monitor has been renamed to ELManager to match its current responsibilities better. * The ELManager is no longer optional in the code (it won't have a nil value under any circumstances). * The support for subscribing for headers was removed as it only worked with WebSockets and contributed significant complexity while bringing only a very minor advantage. * The `--web3-url` parameter has been deprecated in favor of a new `--el` parameter. The new parameter has a reasonable default value and supports specifying a different JWT for each connection. Each connection can also be configured with a different set of responsibilities (e.g. download deposits, validate blocks and/or produce blocks). On the command-line, these properties can be configured through URL properties stored in the #anchor part of the URL. In TOML files, they come with a very natural syntax (althrough the URL scheme is also supported). * The previously scattered EL-related state and logic is now moved to `eth1_monitor.nim` (this module will be renamed to `el_manager.nim` in a follow-up commit). State is assigned properly either to the `ELManager` or the to individual `ELConnection` objects where appropriate. The ELManager executes all Engine API requests against all attached EL nodes, in parallel. It compares their results and if there is a disagreement regarding the validity of a certain payload, this is detected and the beacon node is protected from publishing a block with a potential execution layer consensus bug in it. The BN provides metrics per EL node for the number of successful or failed requests for each type Engine API requests. If an EL node goes offline and connectivity is resoted later, we report the problem and the remedy in edge-triggered fashion. * More progress towards implementing Deneb block production in the VC and comparing the value of blocks produced by the EL and the builder API. * Adds a Makefile target for the zhejiang testnet
2023-03-05 01:40:21 +00:00
responseProcessor.processResponse(m.elConnections, requests, idx)
eliminate fcU/getPayload race condition causing missed proposals (#4800)
2023-04-12 09:33:21 +00:00
template assignNextExpectedPayloadParams() =
# Ensure that there's no race condition window where getPayload's check for
# whether it needs to trigger a new fcU payload, due to cache invalidation,
# falsely suggests that the expected payload matches, and similarly that if
# the fcU fails or times out for other reasons, the expected payload params
# remain synchronized with EL state.
assign(
m.nextExpectedPayloadParams,
some NextExpectedPayloadParams(
headBlockHash: headBlockHash,
safeBlockHash: safeBlockHash,
finalizedBlockHash: finalizedBlockHash,
use fcU V3 in Deneb/Cancun (#5293)
2023-08-15 23:00:35 +00:00
payloadAttributes: payloadAttributesV3))
eliminate fcU/getPayload race condition causing missed proposals (#4800)
2023-04-12 09:33:21 +00:00
Support for driving multiple EL nodes from a single Nimbus BN (#4465) * Support for driving multiple EL nodes from a single Nimbus BN Full list of changes: * Eth1Monitor has been renamed to ELManager to match its current responsibilities better. * The ELManager is no longer optional in the code (it won't have a nil value under any circumstances). * The support for subscribing for headers was removed as it only worked with WebSockets and contributed significant complexity while bringing only a very minor advantage. * The `--web3-url` parameter has been deprecated in favor of a new `--el` parameter. The new parameter has a reasonable default value and supports specifying a different JWT for each connection. Each connection can also be configured with a different set of responsibilities (e.g. download deposits, validate blocks and/or produce blocks). On the command-line, these properties can be configured through URL properties stored in the #anchor part of the URL. In TOML files, they come with a very natural syntax (althrough the URL scheme is also supported). * The previously scattered EL-related state and logic is now moved to `eth1_monitor.nim` (this module will be renamed to `el_manager.nim` in a follow-up commit). State is assigned properly either to the `ELManager` or the to individual `ELConnection` objects where appropriate. The ELManager executes all Engine API requests against all attached EL nodes, in parallel. It compares their results and if there is a disagreement regarding the validity of a certain payload, this is detected and the beacon node is protected from publishing a block with a potential execution layer consensus bug in it. The BN provides metrics per EL node for the number of successful or failed requests for each type Engine API requests. If an EL node goes offline and connectivity is resoted later, we report the problem and the remedy in edge-triggered fashion. * More progress towards implementing Deneb block production in the VC and comparing the value of blocks produced by the EL and the builder API. * Adds a Makefile target for the zhejiang testnet
2023-03-05 01:40:21 +00:00
if responseProcessor.disagreementAlreadyDetected:
return (PayloadExecutionStatus.invalid, none BlockHash)
elif responseProcessor.selectedResponse.isSome:
eliminate fcU/getPayload race condition causing missed proposals (#4800)
2023-04-12 09:33:21 +00:00
assignNextExpectedPayloadParams()
Support for driving multiple EL nodes from a single Nimbus BN (#4465) * Support for driving multiple EL nodes from a single Nimbus BN Full list of changes: * Eth1Monitor has been renamed to ELManager to match its current responsibilities better. * The ELManager is no longer optional in the code (it won't have a nil value under any circumstances). * The support for subscribing for headers was removed as it only worked with WebSockets and contributed significant complexity while bringing only a very minor advantage. * The `--web3-url` parameter has been deprecated in favor of a new `--el` parameter. The new parameter has a reasonable default value and supports specifying a different JWT for each connection. Each connection can also be configured with a different set of responsibilities (e.g. download deposits, validate blocks and/or produce blocks). On the command-line, these properties can be configured through URL properties stored in the #anchor part of the URL. In TOML files, they come with a very natural syntax (althrough the URL scheme is also supported). * The previously scattered EL-related state and logic is now moved to `eth1_monitor.nim` (this module will be renamed to `el_manager.nim` in a follow-up commit). State is assigned properly either to the `ELManager` or the to individual `ELConnection` objects where appropriate. The ELManager executes all Engine API requests against all attached EL nodes, in parallel. It compares their results and if there is a disagreement regarding the validity of a certain payload, this is detected and the beacon node is protected from publishing a block with a potential execution layer consensus bug in it. The BN provides metrics per EL node for the number of successful or failed requests for each type Engine API requests. If an EL node goes offline and connectivity is resoted later, we report the problem and the remedy in edge-triggered fashion. * More progress towards implementing Deneb block production in the VC and comparing the value of blocks produced by the EL and the builder API. * Adds a Makefile target for the zhejiang testnet
2023-03-05 01:40:21 +00:00
return (requests[responseProcessor.selectedResponse.get].read.status,
requests[responseProcessor.selectedResponse.get].read.latestValidHash)
await requestsCompleted or deadline
for idx, req in requests:
if req.completed and req in stillPending:
responseProcessor.processResponse(m.elConnections, requests, idx)
return if responseProcessor.disagreementAlreadyDetected:
(PayloadExecutionStatus.invalid, none BlockHash)
elif responseProcessor.selectedResponse.isSome:
eliminate fcU/getPayload race condition causing missed proposals (#4800)
2023-04-12 09:33:21 +00:00
assignNextExpectedPayloadParams()
Support for driving multiple EL nodes from a single Nimbus BN (#4465) * Support for driving multiple EL nodes from a single Nimbus BN Full list of changes: * Eth1Monitor has been renamed to ELManager to match its current responsibilities better. * The ELManager is no longer optional in the code (it won't have a nil value under any circumstances). * The support for subscribing for headers was removed as it only worked with WebSockets and contributed significant complexity while bringing only a very minor advantage. * The `--web3-url` parameter has been deprecated in favor of a new `--el` parameter. The new parameter has a reasonable default value and supports specifying a different JWT for each connection. Each connection can also be configured with a different set of responsibilities (e.g. download deposits, validate blocks and/or produce blocks). On the command-line, these properties can be configured through URL properties stored in the #anchor part of the URL. In TOML files, they come with a very natural syntax (althrough the URL scheme is also supported). * The previously scattered EL-related state and logic is now moved to `eth1_monitor.nim` (this module will be renamed to `el_manager.nim` in a follow-up commit). State is assigned properly either to the `ELManager` or the to individual `ELConnection` objects where appropriate. The ELManager executes all Engine API requests against all attached EL nodes, in parallel. It compares their results and if there is a disagreement regarding the validity of a certain payload, this is detected and the beacon node is protected from publishing a block with a potential execution layer consensus bug in it. The BN provides metrics per EL node for the number of successful or failed requests for each type Engine API requests. If an EL node goes offline and connectivity is resoted later, we report the problem and the remedy in edge-triggered fashion. * More progress towards implementing Deneb block production in the VC and comparing the value of blocks produced by the EL and the builder API. * Adds a Makefile target for the zhejiang testnet
2023-03-05 01:40:21 +00:00
(requests[responseProcessor.selectedResponse.get].read.status,
requests[responseProcessor.selectedResponse.get].read.latestValidHash)
else:
(PayloadExecutionStatus.syncing, none BlockHash)
# TODO can't be defined within exchangeConfigWithSingleEL
simplify ELMonitor fcU payload attributes handling (#4696)
2023-03-06 16:19:15 +00:00
func `==`(x, y: Quantity): bool {.borrow.}
Support for driving multiple EL nodes from a single Nimbus BN (#4465) * Support for driving multiple EL nodes from a single Nimbus BN Full list of changes: * Eth1Monitor has been renamed to ELManager to match its current responsibilities better. * The ELManager is no longer optional in the code (it won't have a nil value under any circumstances). * The support for subscribing for headers was removed as it only worked with WebSockets and contributed significant complexity while bringing only a very minor advantage. * The `--web3-url` parameter has been deprecated in favor of a new `--el` parameter. The new parameter has a reasonable default value and supports specifying a different JWT for each connection. Each connection can also be configured with a different set of responsibilities (e.g. download deposits, validate blocks and/or produce blocks). On the command-line, these properties can be configured through URL properties stored in the #anchor part of the URL. In TOML files, they come with a very natural syntax (althrough the URL scheme is also supported). * The previously scattered EL-related state and logic is now moved to `eth1_monitor.nim` (this module will be renamed to `el_manager.nim` in a follow-up commit). State is assigned properly either to the `ELManager` or the to individual `ELConnection` objects where appropriate. The ELManager executes all Engine API requests against all attached EL nodes, in parallel. It compares their results and if there is a disagreement regarding the validity of a certain payload, this is detected and the beacon node is protected from publishing a block with a potential execution layer consensus bug in it. The BN provides metrics per EL node for the number of successful or failed requests for each type Engine API requests. If an EL node goes offline and connectivity is resoted later, we report the problem and the remedy in edge-triggered fashion. * More progress towards implementing Deneb block production in the VC and comparing the value of blocks produced by the EL and the builder API. * Adds a Makefile target for the zhejiang testnet
2023-03-05 01:40:21 +00:00
proc exchangeConfigWithSingleEL(m: ELManager, connection: ELConnection) {.async.} =
let rpcClient = await connection.connectedRpcClient()
if m.eth1Network.isSome and
connection.etcStatus == EtcStatus.notExchangedYet:
Avoid potential stalls in the Eth1 monitor code due to missing timeouts (#3905) * Avoid potential stalls in the Eth1 monitor code due to missing timeouts * 1 second timeout for exchange-configuration
2022-07-25 19:23:02 +00:00
try:
Support for driving multiple EL nodes from a single Nimbus BN (#4465) * Support for driving multiple EL nodes from a single Nimbus BN Full list of changes: * Eth1Monitor has been renamed to ELManager to match its current responsibilities better. * The ELManager is no longer optional in the code (it won't have a nil value under any circumstances). * The support for subscribing for headers was removed as it only worked with WebSockets and contributed significant complexity while bringing only a very minor advantage. * The `--web3-url` parameter has been deprecated in favor of a new `--el` parameter. The new parameter has a reasonable default value and supports specifying a different JWT for each connection. Each connection can also be configured with a different set of responsibilities (e.g. download deposits, validate blocks and/or produce blocks). On the command-line, these properties can be configured through URL properties stored in the #anchor part of the URL. In TOML files, they come with a very natural syntax (althrough the URL scheme is also supported). * The previously scattered EL-related state and logic is now moved to `eth1_monitor.nim` (this module will be renamed to `el_manager.nim` in a follow-up commit). State is assigned properly either to the `ELManager` or the to individual `ELConnection` objects where appropriate. The ELManager executes all Engine API requests against all attached EL nodes, in parallel. It compares their results and if there is a disagreement regarding the validity of a certain payload, this is detected and the beacon node is protected from publishing a block with a potential execution layer consensus bug in it. The BN provides metrics per EL node for the number of successful or failed requests for each type Engine API requests. If an EL node goes offline and connectivity is resoted later, we report the problem and the remedy in edge-triggered fashion. * More progress towards implementing Deneb block production in the VC and comparing the value of blocks produced by the EL and the builder API. * Adds a Makefile target for the zhejiang testnet
2023-03-05 01:40:21 +00:00
let
providerChain =
More metrics for the EL monitor (#4707) * `engine_api_response_time` provides a histogram for the Engine API response times for each unique pair ot URL and request type. * All engine API requests are now tracked Other changes: The client will no longer exit on start-up if it fails to connect to a properly configured EL node.
2023-03-09 17:29:04 +00:00
connection.trackedRequestWithTimeout(
"chainId",
rpcClient.eth_chainId(),
web3RequestsTimeout)
Support for driving multiple EL nodes from a single Nimbus BN (#4465) * Support for driving multiple EL nodes from a single Nimbus BN Full list of changes: * Eth1Monitor has been renamed to ELManager to match its current responsibilities better. * The ELManager is no longer optional in the code (it won't have a nil value under any circumstances). * The support for subscribing for headers was removed as it only worked with WebSockets and contributed significant complexity while bringing only a very minor advantage. * The `--web3-url` parameter has been deprecated in favor of a new `--el` parameter. The new parameter has a reasonable default value and supports specifying a different JWT for each connection. Each connection can also be configured with a different set of responsibilities (e.g. download deposits, validate blocks and/or produce blocks). On the command-line, these properties can be configured through URL properties stored in the #anchor part of the URL. In TOML files, they come with a very natural syntax (althrough the URL scheme is also supported). * The previously scattered EL-related state and logic is now moved to `eth1_monitor.nim` (this module will be renamed to `el_manager.nim` in a follow-up commit). State is assigned properly either to the `ELManager` or the to individual `ELConnection` objects where appropriate. The ELManager executes all Engine API requests against all attached EL nodes, in parallel. It compares their results and if there is a disagreement regarding the validity of a certain payload, this is detected and the beacon node is protected from publishing a block with a potential execution layer consensus bug in it. The BN provides metrics per EL node for the number of successful or failed requests for each type Engine API requests. If an EL node goes offline and connectivity is resoted later, we report the problem and the remedy in edge-triggered fashion. * More progress towards implementing Deneb block production in the VC and comparing the value of blocks produced by the EL and the builder API. * Adds a Makefile target for the zhejiang testnet
2023-03-05 01:40:21 +00:00
# https://eips.ethereum.org/EIPS/eip-155#list-of-chain-ids
expectedChain = case m.eth1Network.get
of mainnet: 1.Quantity
of ropsten: 3.Quantity
of rinkeby: 4.Quantity
of goerli: 5.Quantity
of sepolia: 11155111.Quantity # https://chainid.network/
Add metadata for the Holesky network (#5337) * Add metadata for the Holesky network * Add copyright banner to the new Nim module * Working version * Bump Chronos to fix downloading from Github * Add checksum check of the downloaded file * Clean up debugging code and obsolete imports
2023-09-08 05:53:27 +00:00
of holesky: 17000.Quantity
Support for driving multiple EL nodes from a single Nimbus BN (#4465) * Support for driving multiple EL nodes from a single Nimbus BN Full list of changes: * Eth1Monitor has been renamed to ELManager to match its current responsibilities better. * The ELManager is no longer optional in the code (it won't have a nil value under any circumstances). * The support for subscribing for headers was removed as it only worked with WebSockets and contributed significant complexity while bringing only a very minor advantage. * The `--web3-url` parameter has been deprecated in favor of a new `--el` parameter. The new parameter has a reasonable default value and supports specifying a different JWT for each connection. Each connection can also be configured with a different set of responsibilities (e.g. download deposits, validate blocks and/or produce blocks). On the command-line, these properties can be configured through URL properties stored in the #anchor part of the URL. In TOML files, they come with a very natural syntax (althrough the URL scheme is also supported). * The previously scattered EL-related state and logic is now moved to `eth1_monitor.nim` (this module will be renamed to `el_manager.nim` in a follow-up commit). State is assigned properly either to the `ELManager` or the to individual `ELConnection` objects where appropriate. The ELManager executes all Engine API requests against all attached EL nodes, in parallel. It compares their results and if there is a disagreement regarding the validity of a certain payload, this is detected and the beacon node is protected from publishing a block with a potential execution layer consensus bug in it. The BN provides metrics per EL node for the number of successful or failed requests for each type Engine API requests. If an EL node goes offline and connectivity is resoted later, we report the problem and the remedy in edge-triggered fashion. * More progress towards implementing Deneb block production in the VC and comparing the value of blocks produced by the EL and the builder API. * Adds a Makefile target for the zhejiang testnet
2023-03-05 01:40:21 +00:00
if expectedChain != providerChain:
warn "The specified EL client is connected to a different chain",
url = connection.engineUrl,
expectedChain = distinctBase(expectedChain),
actualChain = distinctBase(providerChain)
connection.etcStatus = EtcStatus.mismatch
return
except CatchableError as exc:
# Typically because it's not synced through EIP-155, assuming this Web3
# endpoint has been otherwise working.
debug "Failed to obtain eth_chainId",
error = exc.msg
ignore EL exchangeTransitionConfiguration results (#4831)
2023-04-19 12:16:50 +00:00
connection.etcStatus = EtcStatus.match
update engine API URLs from v1.0.0-beta.2 to beta.3 (#4828)
2023-04-17 20:11:28 +00:00
# https://github.com/ethereum/execution-apis/blob/v1.0.0-beta.3/src/engine/paris.md#engine_exchangetransitionconfigurationv1
Support for driving multiple EL nodes from a single Nimbus BN (#4465) * Support for driving multiple EL nodes from a single Nimbus BN Full list of changes: * Eth1Monitor has been renamed to ELManager to match its current responsibilities better. * The ELManager is no longer optional in the code (it won't have a nil value under any circumstances). * The support for subscribing for headers was removed as it only worked with WebSockets and contributed significant complexity while bringing only a very minor advantage. * The `--web3-url` parameter has been deprecated in favor of a new `--el` parameter. The new parameter has a reasonable default value and supports specifying a different JWT for each connection. Each connection can also be configured with a different set of responsibilities (e.g. download deposits, validate blocks and/or produce blocks). On the command-line, these properties can be configured through URL properties stored in the #anchor part of the URL. In TOML files, they come with a very natural syntax (althrough the URL scheme is also supported). * The previously scattered EL-related state and logic is now moved to `eth1_monitor.nim` (this module will be renamed to `el_manager.nim` in a follow-up commit). State is assigned properly either to the `ELManager` or the to individual `ELConnection` objects where appropriate. The ELManager executes all Engine API requests against all attached EL nodes, in parallel. It compares their results and if there is a disagreement regarding the validity of a certain payload, this is detected and the beacon node is protected from publishing a block with a potential execution layer consensus bug in it. The BN provides metrics per EL node for the number of successful or failed requests for each type Engine API requests. If an EL node goes offline and connectivity is resoted later, we report the problem and the remedy in edge-triggered fashion. * More progress towards implementing Deneb block production in the VC and comparing the value of blocks produced by the EL and the builder API. * Adds a Makefile target for the zhejiang testnet
2023-03-05 01:40:21 +00:00
let
ourConf = TransitionConfigurationV1(
terminalTotalDifficulty: m.eth1Chain.cfg.TERMINAL_TOTAL_DIFFICULTY,
terminalBlockHash: m.eth1Chain.cfg.TERMINAL_BLOCK_HASH,
terminalBlockNumber: Quantity 0)
ignore EL exchangeTransitionConfiguration results (#4831)
2023-04-19 12:16:50 +00:00
try:
discard connection.trackedRequestWithTimeout(
"exchangeTransitionConfiguration",
rpcClient.engine_exchangeTransitionConfigurationV1(ourConf),
timeout = 1.seconds)
except CatchableError as err:
warn "Failed to exchange transition configuration",
url = connection.engineUrl, err = err.msg
Bellatrix TTD detection (#3745) * Bellatrix TTD detection * Update beacon_chain/eth1/eth1_monitor.nim Co-authored-by: Etan Kissling <etan@status.im> * Update beacon_chain/nimbus_beacon_node.nim Co-authored-by: tersec <tersec@users.noreply.github.com> Co-authored-by: Etan Kissling <etan@status.im> Co-authored-by: tersec <tersec@users.noreply.github.com>
2022-06-15 02:38:27 +00:00
Support for driving multiple EL nodes from a single Nimbus BN (#4465) * Support for driving multiple EL nodes from a single Nimbus BN Full list of changes: * Eth1Monitor has been renamed to ELManager to match its current responsibilities better. * The ELManager is no longer optional in the code (it won't have a nil value under any circumstances). * The support for subscribing for headers was removed as it only worked with WebSockets and contributed significant complexity while bringing only a very minor advantage. * The `--web3-url` parameter has been deprecated in favor of a new `--el` parameter. The new parameter has a reasonable default value and supports specifying a different JWT for each connection. Each connection can also be configured with a different set of responsibilities (e.g. download deposits, validate blocks and/or produce blocks). On the command-line, these properties can be configured through URL properties stored in the #anchor part of the URL. In TOML files, they come with a very natural syntax (althrough the URL scheme is also supported). * The previously scattered EL-related state and logic is now moved to `eth1_monitor.nim` (this module will be renamed to `el_manager.nim` in a follow-up commit). State is assigned properly either to the `ELManager` or the to individual `ELConnection` objects where appropriate. The ELManager executes all Engine API requests against all attached EL nodes, in parallel. It compares their results and if there is a disagreement regarding the validity of a certain payload, this is detected and the beacon node is protected from publishing a block with a potential execution layer consensus bug in it. The BN provides metrics per EL node for the number of successful or failed requests for each type Engine API requests. If an EL node goes offline and connectivity is resoted later, we report the problem and the remedy in edge-triggered fashion. * More progress towards implementing Deneb block production in the VC and comparing the value of blocks produced by the EL and the builder API. * Adds a Makefile target for the zhejiang testnet
2023-03-05 01:40:21 +00:00
proc exchangeTransitionConfiguration*(m: ELManager) {.async.} =
if m.elConnections.len == 0:
return
let
deadline = sleepAsync(3.seconds)
requests = m.elConnections.mapIt(m.exchangeConfigWithSingleEL(it))
requestsCompleted = allFutures(requests)
await requestsCompleted or deadline
More forceful EL connection resetting (fixes #4424) (#4708)
2023-03-09 23:41:28 +00:00
var cancelled = 0
Support for driving multiple EL nodes from a single Nimbus BN (#4465) * Support for driving multiple EL nodes from a single Nimbus BN Full list of changes: * Eth1Monitor has been renamed to ELManager to match its current responsibilities better. * The ELManager is no longer optional in the code (it won't have a nil value under any circumstances). * The support for subscribing for headers was removed as it only worked with WebSockets and contributed significant complexity while bringing only a very minor advantage. * The `--web3-url` parameter has been deprecated in favor of a new `--el` parameter. The new parameter has a reasonable default value and supports specifying a different JWT for each connection. Each connection can also be configured with a different set of responsibilities (e.g. download deposits, validate blocks and/or produce blocks). On the command-line, these properties can be configured through URL properties stored in the #anchor part of the URL. In TOML files, they come with a very natural syntax (althrough the URL scheme is also supported). * The previously scattered EL-related state and logic is now moved to `eth1_monitor.nim` (this module will be renamed to `el_manager.nim` in a follow-up commit). State is assigned properly either to the `ELManager` or the to individual `ELConnection` objects where appropriate. The ELManager executes all Engine API requests against all attached EL nodes, in parallel. It compares their results and if there is a disagreement regarding the validity of a certain payload, this is detected and the beacon node is protected from publishing a block with a potential execution layer consensus bug in it. The BN provides metrics per EL node for the number of successful or failed requests for each type Engine API requests. If an EL node goes offline and connectivity is resoted later, we report the problem and the remedy in edge-triggered fashion. * More progress towards implementing Deneb block production in the VC and comparing the value of blocks produced by the EL and the builder API. * Adds a Makefile target for the zhejiang testnet
2023-03-05 01:40:21 +00:00
for idx, req in requests:
if not req.finished:
BN+LC+SN: Fix cancellation deprecate warnings. (#5455) * Fix deprecation cancel() warnings in SN, BN, LC. * Fix missing import.
2023-09-22 11:06:27 +00:00
req.cancelSoon()
More forceful EL connection resetting (fixes #4424) (#4708)
2023-03-09 23:41:28 +00:00
inc cancelled
if cancelled == requests.len:
warn "Failed to exchange configuration with the configured EL end-points"
Support for driving multiple EL nodes from a single Nimbus BN (#4465) * Support for driving multiple EL nodes from a single Nimbus BN Full list of changes: * Eth1Monitor has been renamed to ELManager to match its current responsibilities better. * The ELManager is no longer optional in the code (it won't have a nil value under any circumstances). * The support for subscribing for headers was removed as it only worked with WebSockets and contributed significant complexity while bringing only a very minor advantage. * The `--web3-url` parameter has been deprecated in favor of a new `--el` parameter. The new parameter has a reasonable default value and supports specifying a different JWT for each connection. Each connection can also be configured with a different set of responsibilities (e.g. download deposits, validate blocks and/or produce blocks). On the command-line, these properties can be configured through URL properties stored in the #anchor part of the URL. In TOML files, they come with a very natural syntax (althrough the URL scheme is also supported). * The previously scattered EL-related state and logic is now moved to `eth1_monitor.nim` (this module will be renamed to `el_manager.nim` in a follow-up commit). State is assigned properly either to the `ELManager` or the to individual `ELConnection` objects where appropriate. The ELManager executes all Engine API requests against all attached EL nodes, in parallel. It compares their results and if there is a disagreement regarding the validity of a certain payload, this is detected and the beacon node is protected from publishing a block with a potential execution layer consensus bug in it. The BN provides metrics per EL node for the number of successful or failed requests for each type Engine API requests. If an EL node goes offline and connectivity is resoted later, we report the problem and the remedy in edge-triggered fashion. * More progress towards implementing Deneb block production in the VC and comparing the value of blocks produced by the EL and the builder API. * Adds a Makefile target for the zhejiang testnet
2023-03-05 01:40:21 +00:00
Further reduce the number of RPC requests; Significantly faster ETH1 syncing
2020-11-03 01:21:07 +00:00
template readJsonField(j: JsonNode, fieldName: string, ValueType: type): untyped =
Switch to monitoring of Blocks; More accurate genesis detection and faster Eth1 syncing
2020-06-27 12:01:19 +00:00
var res: ValueType
fromJson(j[fieldName], fieldName, res)
res
The deposit contract uses dynamic bytes for most of its event properties https://github.com/ethereum/consensus-specs/blob/dev/solidity_deposit_contract/deposit_contract.sol#L19-L24
2021-10-14 12:33:55 +00:00
template init[N: static int](T: type DynamicBytes[N, N]): T =
T newSeq[byte](N)
More metrics for the EL monitor (#4707) * `engine_api_response_time` provides a histogram for the Engine API response times for each unique pair ot URL and request type. * All engine API requests are now tracked Other changes: The client will no longer exit on start-up if it fails to connect to a properly configured EL node.
2023-03-09 17:29:04 +00:00
proc fetchTimestamp(connection: ELConnection,
rpcClient: RpcClient,
blk: Eth1Block) {.async.} =
debug "Fetching block timestamp", blockNum = blk.number
let web3block = connection.trackedRequestWithTimeout(
"getBlockByHash",
Support for driving multiple EL nodes from a single Nimbus BN (#4465) * Support for driving multiple EL nodes from a single Nimbus BN Full list of changes: * Eth1Monitor has been renamed to ELManager to match its current responsibilities better. * The ELManager is no longer optional in the code (it won't have a nil value under any circumstances). * The support for subscribing for headers was removed as it only worked with WebSockets and contributed significant complexity while bringing only a very minor advantage. * The `--web3-url` parameter has been deprecated in favor of a new `--el` parameter. The new parameter has a reasonable default value and supports specifying a different JWT for each connection. Each connection can also be configured with a different set of responsibilities (e.g. download deposits, validate blocks and/or produce blocks). On the command-line, these properties can be configured through URL properties stored in the #anchor part of the URL. In TOML files, they come with a very natural syntax (althrough the URL scheme is also supported). * The previously scattered EL-related state and logic is now moved to `eth1_monitor.nim` (this module will be renamed to `el_manager.nim` in a follow-up commit). State is assigned properly either to the `ELManager` or the to individual `ELConnection` objects where appropriate. The ELManager executes all Engine API requests against all attached EL nodes, in parallel. It compares their results and if there is a disagreement regarding the validity of a certain payload, this is detected and the beacon node is protected from publishing a block with a potential execution layer consensus bug in it. The BN provides metrics per EL node for the number of successful or failed requests for each type Engine API requests. If an EL node goes offline and connectivity is resoted later, we report the problem and the remedy in edge-triggered fashion. * More progress towards implementing Deneb block production in the VC and comparing the value of blocks produced by the EL and the builder API. * Adds a Makefile target for the zhejiang testnet
2023-03-05 01:40:21 +00:00
rpcClient.getBlockByHash(blk.hash.asBlockHash),
web3RequestsTimeout)
More metrics for the EL monitor (#4707) * `engine_api_response_time` provides a histogram for the Engine API response times for each unique pair ot URL and request type. * All engine API requests are now tracked Other changes: The client will no longer exit on start-up if it fails to connect to a properly configured EL node.
2023-03-09 17:29:04 +00:00
Minor post-merge cleanups (#3945) https://github.com/status-im/nimbus-eth2/pull/3944 The use of nested `awaitWithRetries` calls would have resulted in an unexpected number of retries (3x3). We now use regular `await` in outer layer to avoid the problem. https://github.com/status-im/nimbus-eth2/pull/3943 The new code has an invariant that the `headMerkleizer` field in the `Eth1Chain` is always kept in sync with the blocks stored in the chain. This invariant is now enforced better by doing the necessary merkleizer updates in the `Eth1Chain.addBlock` function, in the `Eth1Chain.init` function and in the `Eth1Chain.reset` function.
2022-08-10 12:31:10 +00:00
blk.timestamp = Eth1BlockTimestamp web3block.timestamp
implement terminal-total-difficulty-override; keep kintsugi TTDs consistent (#3118) * implement terminal-total-difficulty-override; keep TTD consistent for m2 scripts/docs * use Option[uint64] instead of uint64
2021-11-25 10:53:31 +00:00
func depositEventsToBlocks(depositsList: JsonNode): seq[Eth1Block] {.
remove `{.raises: [Defect].}` Nim 1.2 compatibility (#5352) In Nim 1.6, `{.raises: [Defect].}` is no longer necessary. Remove it.
2023-08-25 09:29:07 +00:00
raises: [CatchableError].} =
Switch to monitoring of Blocks; More accurate genesis detection and faster Eth1 syncing
2020-06-27 12:01:19 +00:00
if depositsList.kind != JArray:
raise newException(CatchableError,
"Web3 provider didn't return a list of deposit events")
var lastEth1Block: Eth1Block
for logEvent in depositsList:
let
blockNumber = Eth1BlockNumber readJsonField(logEvent, "blockNumber", Quantity)
blockHash = readJsonField(logEvent, "blockHash", BlockHash)
logData = strip0xPrefix(logEvent["data"].getStr)
if lastEth1Block == nil or lastEth1Block.number != blockNumber:
lastEth1Block = Eth1Block(
Minor post-merge cleanups (#3945) https://github.com/status-im/nimbus-eth2/pull/3944 The use of nested `awaitWithRetries` calls would have resulted in an unexpected number of retries (3x3). We now use regular `await` in outer layer to avoid the problem. https://github.com/status-im/nimbus-eth2/pull/3943 The new code has an invariant that the `headMerkleizer` field in the `Eth1Chain` is always kept in sync with the blocks stored in the chain. This invariant is now enforced better by doing the necessary merkleizer updates in the `Eth1Chain.addBlock` function, in the `Eth1Chain.init` function and in the `Eth1Chain.reset` function.
2022-08-10 12:31:10 +00:00
hash: blockHash.asEth2Digest,
number: blockNumber
# The `timestamp` is set in `syncBlockRange` immediately
# after calling this function, because we don't want to
# make this function `async`
)
Switch to monitoring of Blocks; More accurate genesis detection and faster Eth1 syncing
2020-06-27 12:01:19 +00:00
result.add lastEth1Block
var
The deposit contract uses dynamic bytes for most of its event properties https://github.com/ethereum/consensus-specs/blob/dev/solidity_deposit_contract/deposit_contract.sol#L19-L24
2021-10-14 12:33:55 +00:00
pubkey = init PubKeyBytes
withdrawalCredentials = init WithdrawalCredentialsBytes
amount = init Int64LeBytes
signature = init SignatureBytes
index = init Int64LeBytes
Switch to monitoring of Blocks; More accurate genesis detection and faster Eth1 syncing
2020-06-27 12:01:19 +00:00
var offset = 0
offset += decode(logData, offset, pubkey)
offset += decode(logData, offset, withdrawalCredentials)
offset += decode(logData, offset, amount)
offset += decode(logData, offset, signature)
offset += decode(logData, offset, index)
The deposit contract uses dynamic bytes for most of its event properties https://github.com/ethereum/consensus-specs/blob/dev/solidity_deposit_contract/deposit_contract.sol#L19-L24
2021-10-14 12:33:55 +00:00
if pubkey.len != 48 or
withdrawalCredentials.len != 32 or
amount.len != 8 or
signature.len != 96 or
index.len != 8:
raise newException(CorruptDataProvider, "Web3 provider supplied invalid deposit logs")
Fix the corrupted database state on Pyrmont nodes; Add mainnet genesis (#2056) * Handle some web3 timeouts better * Add support for developer .env files * Eth1 improvements; Mainnet genesis state Notable changes: * The deposits table have been removed from the database. The client will no longer process all deposits on start-up. * The network metadata now includes a "state snapshot" of the deposit contract. This allows the client to skip syncing deposits made prior to the snapshot (i.e. genesis). Suitable metadata added for Pyrmont and Mainnet. * The Eth1 monitor won't be started unless there are validators attached to the node. * The genesis detection code is now optional and disabled by default * Bugfix: The client should not produce blocks that will fail validation when it hasn't downloaded the latest deposits yet * Bugfix: Work around the database corruption affecting Pyrmont nodes * Remove metadata for Toledo and Medalla
2020-11-24 21:21:47 +00:00
lastEth1Block.deposits.add DepositData(
The deposit contract uses dynamic bytes for most of its event properties https://github.com/ethereum/consensus-specs/blob/dev/solidity_deposit_contract/deposit_contract.sol#L19-L24
2021-10-14 12:33:55 +00:00
pubkey: ValidatorPubKey.init(pubkey.toArray),
withdrawal_credentials: Eth2Digest(data: withdrawalCredentials.toArray),
amount: bytes_to_uint64(amount.toArray),
signature: ValidatorSig.init(signature.toArray))
Switch to monitoring of Blocks; More accurate genesis detection and faster Eth1 syncing
2020-06-27 12:01:19 +00:00
Reduce the rate of requesting deposits to lower the risk of triggering the Infura rate limits
2020-12-01 11:14:32 +00:00
type
DepositContractDataStatus = enum
Fetched
VerifiedCorrect
DepositRootIncorrect
DepositRootUnavailable
DepositCountIncorrect
DepositCountUnavailable
Further reduce the number of RPC requests; Significantly faster ETH1 syncing
2020-11-03 01:21:07 +00:00
Reduce the rate of requesting deposits to lower the risk of triggering the Infura rate limits
2020-12-01 11:14:32 +00:00
when hasDepositRootChecks:
Fix the corrupted database state on Pyrmont nodes; Add mainnet genesis (#2056) * Handle some web3 timeouts better * Add support for developer .env files * Eth1 improvements; Mainnet genesis state Notable changes: * The deposits table have been removed from the database. The client will no longer process all deposits on start-up. * The network metadata now includes a "state snapshot" of the deposit contract. This allows the client to skip syncing deposits made prior to the snapshot (i.e. genesis). Suitable metadata added for Pyrmont and Mainnet. * The Eth1 monitor won't be started unless there are validators attached to the node. * The genesis detection code is now optional and disabled by default * Bugfix: The client should not produce blocks that will fail validation when it hasn't downloaded the latest deposits yet * Bugfix: Work around the database corruption affecting Pyrmont nodes * Remove metadata for Toledo and Medalla
2020-11-24 21:21:47 +00:00
const
Avoid potential stalls in the Eth1 monitor code due to missing timeouts (#3905) * Avoid potential stalls in the Eth1 monitor code due to missing timeouts * 1 second timeout for exchange-configuration
2022-07-25 19:23:02 +00:00
contractCallTimeout = 60.seconds
Fix the corrupted database state on Pyrmont nodes; Add mainnet genesis (#2056) * Handle some web3 timeouts better * Add support for developer .env files * Eth1 improvements; Mainnet genesis state Notable changes: * The deposits table have been removed from the database. The client will no longer process all deposits on start-up. * The network metadata now includes a "state snapshot" of the deposit contract. This allows the client to skip syncing deposits made prior to the snapshot (i.e. genesis). Suitable metadata added for Pyrmont and Mainnet. * The Eth1 monitor won't be started unless there are validators attached to the node. * The genesis detection code is now optional and disabled by default * Bugfix: The client should not produce blocks that will fail validation when it hasn't downloaded the latest deposits yet * Bugfix: Work around the database corruption affecting Pyrmont nodes * Remove metadata for Toledo and Medalla
2020-11-24 21:21:47 +00:00
More metrics for the EL monitor (#4707) * `engine_api_response_time` provides a histogram for the Engine API response times for each unique pair ot URL and request type. * All engine API requests are now tracked Other changes: The client will no longer exit on start-up if it fails to connect to a properly configured EL node.
2023-03-09 17:29:04 +00:00
proc fetchDepositContractData(connection: ELConnection,
rpcClient: RpcClient,
Support for driving multiple EL nodes from a single Nimbus BN (#4465) * Support for driving multiple EL nodes from a single Nimbus BN Full list of changes: * Eth1Monitor has been renamed to ELManager to match its current responsibilities better. * The ELManager is no longer optional in the code (it won't have a nil value under any circumstances). * The support for subscribing for headers was removed as it only worked with WebSockets and contributed significant complexity while bringing only a very minor advantage. * The `--web3-url` parameter has been deprecated in favor of a new `--el` parameter. The new parameter has a reasonable default value and supports specifying a different JWT for each connection. Each connection can also be configured with a different set of responsibilities (e.g. download deposits, validate blocks and/or produce blocks). On the command-line, these properties can be configured through URL properties stored in the #anchor part of the URL. In TOML files, they come with a very natural syntax (althrough the URL scheme is also supported). * The previously scattered EL-related state and logic is now moved to `eth1_monitor.nim` (this module will be renamed to `el_manager.nim` in a follow-up commit). State is assigned properly either to the `ELManager` or the to individual `ELConnection` objects where appropriate. The ELManager executes all Engine API requests against all attached EL nodes, in parallel. It compares their results and if there is a disagreement regarding the validity of a certain payload, this is detected and the beacon node is protected from publishing a block with a potential execution layer consensus bug in it. The BN provides metrics per EL node for the number of successful or failed requests for each type Engine API requests. If an EL node goes offline and connectivity is resoted later, we report the problem and the remedy in edge-triggered fashion. * More progress towards implementing Deneb block production in the VC and comparing the value of blocks produced by the EL and the builder API. * Adds a Makefile target for the zhejiang testnet
2023-03-05 01:40:21 +00:00
depositContact: Sender[DepositContract],
blk: Eth1Block): Future[DepositContractDataStatus] {.async.} =
Further reduce the number of RPC requests; Significantly faster ETH1 syncing
2020-11-03 01:21:07 +00:00
let
More metrics for the EL monitor (#4707) * `engine_api_response_time` provides a histogram for the Engine API response times for each unique pair ot URL and request type. * All engine API requests are now tracked Other changes: The client will no longer exit on start-up if it fails to connect to a properly configured EL node.
2023-03-09 17:29:04 +00:00
startTime = Moment.now
deadline = sleepAsync(contractCallTimeout)
Support for driving multiple EL nodes from a single Nimbus BN (#4465) * Support for driving multiple EL nodes from a single Nimbus BN Full list of changes: * Eth1Monitor has been renamed to ELManager to match its current responsibilities better. * The ELManager is no longer optional in the code (it won't have a nil value under any circumstances). * The support for subscribing for headers was removed as it only worked with WebSockets and contributed significant complexity while bringing only a very minor advantage. * The `--web3-url` parameter has been deprecated in favor of a new `--el` parameter. The new parameter has a reasonable default value and supports specifying a different JWT for each connection. Each connection can also be configured with a different set of responsibilities (e.g. download deposits, validate blocks and/or produce blocks). On the command-line, these properties can be configured through URL properties stored in the #anchor part of the URL. In TOML files, they come with a very natural syntax (althrough the URL scheme is also supported). * The previously scattered EL-related state and logic is now moved to `eth1_monitor.nim` (this module will be renamed to `el_manager.nim` in a follow-up commit). State is assigned properly either to the `ELManager` or the to individual `ELConnection` objects where appropriate. The ELManager executes all Engine API requests against all attached EL nodes, in parallel. It compares their results and if there is a disagreement regarding the validity of a certain payload, this is detected and the beacon node is protected from publishing a block with a potential execution layer consensus bug in it. The BN provides metrics per EL node for the number of successful or failed requests for each type Engine API requests. If an EL node goes offline and connectivity is resoted later, we report the problem and the remedy in edge-triggered fashion. * More progress towards implementing Deneb block production in the VC and comparing the value of blocks produced by the EL and the builder API. * Adds a Makefile target for the zhejiang testnet
2023-03-05 01:40:21 +00:00
depositRoot = depositContract.get_deposit_root.call(blockNumber = blk.number)
rawCount = depositContract.get_deposit_count.call(blockNumber = blk.number)
Fix #1823
2020-10-21 13:25:53 +00:00
Don't report an 'EL connection degraded' warning when the EL is not synced (#4761)
2023-03-21 22:19:36 +00:00
# We allow failures on these requests becaues the clients
# are expected to prune the state data for historical blocks
More metrics for the EL monitor (#4707) * `engine_api_response_time` provides a histogram for the Engine API response times for each unique pair ot URL and request type. * All engine API requests are now tracked Other changes: The client will no longer exit on start-up if it fails to connect to a properly configured EL node.
2023-03-09 17:29:04 +00:00
connection.trackEngineApiRequest(
Don't report an 'EL connection degraded' warning when the EL is not synced (#4761)
2023-03-21 22:19:36 +00:00
depositRoot, "get_deposit_root", startTime, deadline,
failureAllowed = true)
More metrics for the EL monitor (#4707) * `engine_api_response_time` provides a histogram for the Engine API response times for each unique pair ot URL and request type. * All engine API requests are now tracked Other changes: The client will no longer exit on start-up if it fails to connect to a properly configured EL node.
2023-03-09 17:29:04 +00:00
connection.trackEngineApiRequest(
Don't report an 'EL connection degraded' warning when the EL is not synced (#4761)
2023-03-21 22:19:36 +00:00
rawCount, "get_deposit_count", startTime, deadline,
failureAllowed = true)
More metrics for the EL monitor (#4707) * `engine_api_response_time` provides a histogram for the Engine API response times for each unique pair ot URL and request type. * All engine API requests are now tracked Other changes: The client will no longer exit on start-up if it fails to connect to a properly configured EL node.
2023-03-09 17:29:04 +00:00
Further reduce the number of RPC requests; Significantly faster ETH1 syncing
2020-11-03 01:21:07 +00:00
try:
Fix the corrupted database state on Pyrmont nodes; Add mainnet genesis (#2056) * Handle some web3 timeouts better * Add support for developer .env files * Eth1 improvements; Mainnet genesis state Notable changes: * The deposits table have been removed from the database. The client will no longer process all deposits on start-up. * The network metadata now includes a "state snapshot" of the deposit contract. This allows the client to skip syncing deposits made prior to the snapshot (i.e. genesis). Suitable metadata added for Pyrmont and Mainnet. * The Eth1 monitor won't be started unless there are validators attached to the node. * The genesis detection code is now optional and disabled by default * Bugfix: The client should not produce blocks that will fail validation when it hasn't downloaded the latest deposits yet * Bugfix: Work around the database corruption affecting Pyrmont nodes * Remove metadata for Toledo and Medalla
2020-11-24 21:21:47 +00:00
let fetchedRoot = asEth2Digest(
More metrics for the EL monitor (#4707) * `engine_api_response_time` provides a histogram for the Engine API response times for each unique pair ot URL and request type. * All engine API requests are now tracked Other changes: The client will no longer exit on start-up if it fails to connect to a properly configured EL node.
2023-03-09 17:29:04 +00:00
awaitWithTimeout(depositRoot, deadline))
Minor post-merge cleanups (#3945) https://github.com/status-im/nimbus-eth2/pull/3944 The use of nested `awaitWithRetries` calls would have resulted in an unexpected number of retries (3x3). We now use regular `await` in outer layer to avoid the problem. https://github.com/status-im/nimbus-eth2/pull/3943 The new code has an invariant that the `headMerkleizer` field in the `Eth1Chain` is always kept in sync with the blocks stored in the chain. This invariant is now enforced better by doing the necessary merkleizer updates in the `Eth1Chain.addBlock` function, in the `Eth1Chain.init` function and in the `Eth1Chain.reset` function.
2022-08-10 12:31:10 +00:00
if blk.depositRoot.isZero:
blk.depositRoot = fetchedRoot
Further reduce the number of RPC requests; Significantly faster ETH1 syncing
2020-11-03 01:21:07 +00:00
result = Fetched
Minor post-merge cleanups (#3945) https://github.com/status-im/nimbus-eth2/pull/3944 The use of nested `awaitWithRetries` calls would have resulted in an unexpected number of retries (3x3). We now use regular `await` in outer layer to avoid the problem. https://github.com/status-im/nimbus-eth2/pull/3943 The new code has an invariant that the `headMerkleizer` field in the `Eth1Chain` is always kept in sync with the blocks stored in the chain. This invariant is now enforced better by doing the necessary merkleizer updates in the `Eth1Chain.addBlock` function, in the `Eth1Chain.init` function and in the `Eth1Chain.reset` function.
2022-08-10 12:31:10 +00:00
elif blk.depositRoot == fetchedRoot:
Further reduce the number of RPC requests; Significantly faster ETH1 syncing
2020-11-03 01:21:07 +00:00
result = VerifiedCorrect
else:
result = DepositRootIncorrect
except CatchableError as err:
debug "Failed to fetch deposits root",
blockNumber = blk.number,
err = err.msg
result = DepositRootUnavailable
Fix #1823
2020-10-21 13:25:53 +00:00
Further reduce the number of RPC requests; Significantly faster ETH1 syncing
2020-11-03 01:21:07 +00:00
try:
fix obtaining deposits after connection loss (#3943) * fix obtaining deposits after connection loss When an error occurs during Eth1 deposits import, the already imported blocks are kept while the connection to the EL is re-established. However, the corresponding merkleizer is not persisted, leading to any future deposits no longer being properly imported. This is quite common when syncing a fresh Nimbus instance against an already-synced Geth EL. Fixed by persisting the head merkleizer together with the blocks.
2022-08-09 21:32:34 +00:00
let fetchedCount = bytes_to_uint64(
More metrics for the EL monitor (#4707) * `engine_api_response_time` provides a histogram for the Engine API response times for each unique pair ot URL and request type. * All engine API requests are now tracked Other changes: The client will no longer exit on start-up if it fails to connect to a properly configured EL node.
2023-03-09 17:29:04 +00:00
awaitWithTimeout(rawCount, deadline).toArray)
Minor post-merge cleanups (#3945) https://github.com/status-im/nimbus-eth2/pull/3944 The use of nested `awaitWithRetries` calls would have resulted in an unexpected number of retries (3x3). We now use regular `await` in outer layer to avoid the problem. https://github.com/status-im/nimbus-eth2/pull/3943 The new code has an invariant that the `headMerkleizer` field in the `Eth1Chain` is always kept in sync with the blocks stored in the chain. This invariant is now enforced better by doing the necessary merkleizer updates in the `Eth1Chain.addBlock` function, in the `Eth1Chain.init` function and in the `Eth1Chain.reset` function.
2022-08-10 12:31:10 +00:00
if blk.depositCount == 0:
blk.depositCount = fetchedCount
elif blk.depositCount != fetchedCount:
Further reduce the number of RPC requests; Significantly faster ETH1 syncing
2020-11-03 01:21:07 +00:00
result = DepositCountIncorrect
except CatchableError as err:
debug "Failed to fetch deposits count",
blockNumber = blk.number,
err = err.msg
result = DepositCountUnavailable
Switch to monitoring of Blocks; More accurate genesis detection and faster Eth1 syncing
2020-06-27 12:01:19 +00:00
Add Eth1 deposits simulation to block_sim
2020-12-03 04:30:35 +00:00
proc pruneOldBlocks(chain: var Eth1Chain, depositIndex: uint64) =
Support for obtaining deposit snapshots during trustedNodeSync (#4303) Other changes: * More optimal search for TTD block. * Add timeouts to all REST requests during trusted node sync. Fixes #4037 * Removed support for storing a deposit snapshot in the network metadata.
2022-12-07 10:24:51 +00:00
## Called on block finalization to delete old and now redundant data.
Add Eth1 deposits simulation to block_sim
2020-12-03 04:30:35 +00:00
let initialChunks = chain.finalizedDepositsMerkleizer.getChunkCount
Fix the corrupted database state on Pyrmont nodes; Add mainnet genesis (#2056) * Handle some web3 timeouts better * Add support for developer .env files * Eth1 improvements; Mainnet genesis state Notable changes: * The deposits table have been removed from the database. The client will no longer process all deposits on start-up. * The network metadata now includes a "state snapshot" of the deposit contract. This allows the client to skip syncing deposits made prior to the snapshot (i.e. genesis). Suitable metadata added for Pyrmont and Mainnet. * The Eth1 monitor won't be started unless there are validators attached to the node. * The genesis detection code is now optional and disabled by default * Bugfix: The client should not produce blocks that will fail validation when it hasn't downloaded the latest deposits yet * Bugfix: Work around the database corruption affecting Pyrmont nodes * Remove metadata for Toledo and Medalla
2020-11-24 21:21:47 +00:00
var lastBlock: Eth1Block
Add Eth1 deposits simulation to block_sim
2020-12-03 04:30:35 +00:00
while chain.blocks.len > 0:
let blk = chain.blocks.peekFirst
Minor post-merge cleanups (#3945) https://github.com/status-im/nimbus-eth2/pull/3944 The use of nested `awaitWithRetries` calls would have resulted in an unexpected number of retries (3x3). We now use regular `await` in outer layer to avoid the problem. https://github.com/status-im/nimbus-eth2/pull/3943 The new code has an invariant that the `headMerkleizer` field in the `Eth1Chain` is always kept in sync with the blocks stored in the chain. This invariant is now enforced better by doing the necessary merkleizer updates in the `Eth1Chain.addBlock` function, in the `Eth1Chain.init` function and in the `Eth1Chain.reset` function.
2022-08-10 12:31:10 +00:00
if blk.depositCount >= depositIndex:
Fix the corrupted database state on Pyrmont nodes; Add mainnet genesis (#2056) * Handle some web3 timeouts better * Add support for developer .env files * Eth1 improvements; Mainnet genesis state Notable changes: * The deposits table have been removed from the database. The client will no longer process all deposits on start-up. * The network metadata now includes a "state snapshot" of the deposit contract. This allows the client to skip syncing deposits made prior to the snapshot (i.e. genesis). Suitable metadata added for Pyrmont and Mainnet. * The Eth1 monitor won't be started unless there are validators attached to the node. * The genesis detection code is now optional and disabled by default * Bugfix: The client should not produce blocks that will fail validation when it hasn't downloaded the latest deposits yet * Bugfix: Work around the database corruption affecting Pyrmont nodes * Remove metadata for Toledo and Medalla
2020-11-24 21:21:47 +00:00
break
else:
for deposit in blk.deposits:
Add Eth1 deposits simulation to block_sim
2020-12-03 04:30:35 +00:00
chain.finalizedDepositsMerkleizer.addChunk hash_tree_root(deposit).data
chain.popFirst()
Fix the corrupted database state on Pyrmont nodes; Add mainnet genesis (#2056) * Handle some web3 timeouts better * Add support for developer .env files * Eth1 improvements; Mainnet genesis state Notable changes: * The deposits table have been removed from the database. The client will no longer process all deposits on start-up. * The network metadata now includes a "state snapshot" of the deposit contract. This allows the client to skip syncing deposits made prior to the snapshot (i.e. genesis). Suitable metadata added for Pyrmont and Mainnet. * The Eth1 monitor won't be started unless there are validators attached to the node. * The genesis detection code is now optional and disabled by default * Bugfix: The client should not produce blocks that will fail validation when it hasn't downloaded the latest deposits yet * Bugfix: Work around the database corruption affecting Pyrmont nodes * Remove metadata for Toledo and Medalla
2020-11-24 21:21:47 +00:00
lastBlock = blk
Add Eth1 deposits simulation to block_sim
2020-12-03 04:30:35 +00:00
if chain.finalizedDepositsMerkleizer.getChunkCount > initialChunks:
Minor post-merge cleanups (#3945) https://github.com/status-im/nimbus-eth2/pull/3944 The use of nested `awaitWithRetries` calls would have resulted in an unexpected number of retries (3x3). We now use regular `await` in outer layer to avoid the problem. https://github.com/status-im/nimbus-eth2/pull/3943 The new code has an invariant that the `headMerkleizer` field in the `Eth1Chain` is always kept in sync with the blocks stored in the chain. This invariant is now enforced better by doing the necessary merkleizer updates in the `Eth1Chain.addBlock` function, in the `Eth1Chain.init` function and in the `Eth1Chain.reset` function.
2022-08-10 12:31:10 +00:00
chain.finalizedBlockHash = lastBlock.hash
Support for obtaining deposit snapshots during trustedNodeSync (#4303) Other changes: * More optimal search for TTD block. * Add timeouts to all REST requests during trusted node sync. Fixes #4037 * Removed support for storing a deposit snapshot in the network metadata.
2022-12-07 10:24:51 +00:00
chain.db.putDepositTreeSnapshot DepositTreeSnapshot(
Minor post-merge cleanups (#3945) https://github.com/status-im/nimbus-eth2/pull/3944 The use of nested `awaitWithRetries` calls would have resulted in an unexpected number of retries (3x3). We now use regular `await` in outer layer to avoid the problem. https://github.com/status-im/nimbus-eth2/pull/3943 The new code has an invariant that the `headMerkleizer` field in the `Eth1Chain` is always kept in sync with the blocks stored in the chain. This invariant is now enforced better by doing the necessary merkleizer updates in the `Eth1Chain.addBlock` function, in the `Eth1Chain.init` function and in the `Eth1Chain.reset` function.
2022-08-10 12:31:10 +00:00
eth1Block: lastBlock.hash,
Support for obtaining deposit snapshots during trustedNodeSync (#4303) Other changes: * More optimal search for TTD block. * Add timeouts to all REST requests during trusted node sync. Fixes #4037 * Removed support for storing a deposit snapshot in the network metadata.
2022-12-07 10:24:51 +00:00
depositContractState: chain.finalizedDepositsMerkleizer.toDepositContractState,
Local sim impovements (#4551) * Local sim impovements * Added support for running Capella and EIP-4844 simulations by downloading the correct version of Geth. * Added support for using Nimbus remote signer and Web3Signer. Use 2 out of 3 threshold signing configuration in the mainnet configuration and regular remote signing in the minimal one. * The local testnet simulation can now use a payload builder. This is currently not activated in CI due to lack of automated procedures for installing third-party relays or builders. You are adviced to use mergemock for now, but for most realistic results, we can create a simple builder based on the nimbus-eth1 codebase that will be able to propose transactions from the regular network mempool. * Start the simulation from a merged state. This would allow us to start removing pre-merge functionality such as the gossip subsciption logic. The commit also removes the merge-forcing hack installed after the TTD removal. * Consolidate all the tools used in the local simulation into a single `ncli_testnet` binary.
2023-02-23 02:10:07 +00:00
blockHeight: lastBlock.number)
Fix the corrupted database state on Pyrmont nodes; Add mainnet genesis (#2056) * Handle some web3 timeouts better * Add support for developer .env files * Eth1 improvements; Mainnet genesis state Notable changes: * The deposits table have been removed from the database. The client will no longer process all deposits on start-up. * The network metadata now includes a "state snapshot" of the deposit contract. This allows the client to skip syncing deposits made prior to the snapshot (i.e. genesis). Suitable metadata added for Pyrmont and Mainnet. * The Eth1 monitor won't be started unless there are validators attached to the node. * The genesis detection code is now optional and disabled by default * Bugfix: The client should not produce blocks that will fail validation when it hasn't downloaded the latest deposits yet * Bugfix: Work around the database corruption affecting Pyrmont nodes * Remove metadata for Toledo and Medalla
2020-11-24 21:21:47 +00:00
Add some useful Eth1 monitor metrics
2020-12-09 22:44:59 +00:00
eth1_finalized_head.set lastBlock.number.toGaugeValue
Minor post-merge cleanups (#3945) https://github.com/status-im/nimbus-eth2/pull/3944 The use of nested `awaitWithRetries` calls would have resulted in an unexpected number of retries (3x3). We now use regular `await` in outer layer to avoid the problem. https://github.com/status-im/nimbus-eth2/pull/3943 The new code has an invariant that the `headMerkleizer` field in the `Eth1Chain` is always kept in sync with the blocks stored in the chain. This invariant is now enforced better by doing the necessary merkleizer updates in the `Eth1Chain.addBlock` function, in the `Eth1Chain.init` function and in the `Eth1Chain.reset` function.
2022-08-10 12:31:10 +00:00
eth1_finalized_deposits.set lastBlock.depositCount.toGaugeValue
Add some useful Eth1 monitor metrics
2020-12-09 22:44:59 +00:00
Less risk for Eth1-induced delays in block proposal
2020-11-30 23:59:35 +00:00
debug "Eth1 blocks pruned",
Minor post-merge cleanups (#3945) https://github.com/status-im/nimbus-eth2/pull/3944 The use of nested `awaitWithRetries` calls would have resulted in an unexpected number of retries (3x3). We now use regular `await` in outer layer to avoid the problem. https://github.com/status-im/nimbus-eth2/pull/3943 The new code has an invariant that the `headMerkleizer` field in the `Eth1Chain` is always kept in sync with the blocks stored in the chain. This invariant is now enforced better by doing the necessary merkleizer updates in the `Eth1Chain.addBlock` function, in the `Eth1Chain.init` function and in the `Eth1Chain.reset` function.
2022-08-10 12:31:10 +00:00
newTailBlock = lastBlock.hash,
depositsCount = lastBlock.depositCount
Less risk for Eth1-induced delays in block proposal
2020-11-30 23:59:35 +00:00
implement terminal-total-difficulty-override; keep kintsugi TTDs consistent (#3118) * implement terminal-total-difficulty-override; keep TTD consistent for m2 scripts/docs * use Option[uint64] instead of uint64
2021-11-25 10:53:31 +00:00
func advanceMerkleizer(chain: Eth1Chain,
Fix the corrupted database state on Pyrmont nodes; Add mainnet genesis (#2056) * Handle some web3 timeouts better * Add support for developer .env files * Eth1 improvements; Mainnet genesis state Notable changes: * The deposits table have been removed from the database. The client will no longer process all deposits on start-up. * The network metadata now includes a "state snapshot" of the deposit contract. This allows the client to skip syncing deposits made prior to the snapshot (i.e. genesis). Suitable metadata added for Pyrmont and Mainnet. * The Eth1 monitor won't be started unless there are validators attached to the node. * The genesis detection code is now optional and disabled by default * Bugfix: The client should not produce blocks that will fail validation when it hasn't downloaded the latest deposits yet * Bugfix: Work around the database corruption affecting Pyrmont nodes * Remove metadata for Toledo and Medalla
2020-11-24 21:21:47 +00:00
merkleizer: var DepositsMerkleizer,
depositIndex: uint64): bool =
Add Eth1 deposits simulation to block_sim
2020-12-03 04:30:35 +00:00
if chain.blocks.len == 0:
Fix the corrupted database state on Pyrmont nodes; Add mainnet genesis (#2056) * Handle some web3 timeouts better * Add support for developer .env files * Eth1 improvements; Mainnet genesis state Notable changes: * The deposits table have been removed from the database. The client will no longer process all deposits on start-up. * The network metadata now includes a "state snapshot" of the deposit contract. This allows the client to skip syncing deposits made prior to the snapshot (i.e. genesis). Suitable metadata added for Pyrmont and Mainnet. * The Eth1 monitor won't be started unless there are validators attached to the node. * The genesis detection code is now optional and disabled by default * Bugfix: The client should not produce blocks that will fail validation when it hasn't downloaded the latest deposits yet * Bugfix: Work around the database corruption affecting Pyrmont nodes * Remove metadata for Toledo and Medalla
2020-11-24 21:21:47 +00:00
return depositIndex == merkleizer.getChunkCount
Minor post-merge cleanups (#3945) https://github.com/status-im/nimbus-eth2/pull/3944 The use of nested `awaitWithRetries` calls would have resulted in an unexpected number of retries (3x3). We now use regular `await` in outer layer to avoid the problem. https://github.com/status-im/nimbus-eth2/pull/3943 The new code has an invariant that the `headMerkleizer` field in the `Eth1Chain` is always kept in sync with the blocks stored in the chain. This invariant is now enforced better by doing the necessary merkleizer updates in the `Eth1Chain.addBlock` function, in the `Eth1Chain.init` function and in the `Eth1Chain.reset` function.
2022-08-10 12:31:10 +00:00
if chain.blocks.peekLast.depositCount < depositIndex:
Fix the corrupted database state on Pyrmont nodes; Add mainnet genesis (#2056) * Handle some web3 timeouts better * Add support for developer .env files * Eth1 improvements; Mainnet genesis state Notable changes: * The deposits table have been removed from the database. The client will no longer process all deposits on start-up. * The network metadata now includes a "state snapshot" of the deposit contract. This allows the client to skip syncing deposits made prior to the snapshot (i.e. genesis). Suitable metadata added for Pyrmont and Mainnet. * The Eth1 monitor won't be started unless there are validators attached to the node. * The genesis detection code is now optional and disabled by default * Bugfix: The client should not produce blocks that will fail validation when it hasn't downloaded the latest deposits yet * Bugfix: Work around the database corruption affecting Pyrmont nodes * Remove metadata for Toledo and Medalla
2020-11-24 21:21:47 +00:00
return false
let
Add Eth1 deposits simulation to block_sim
2020-12-03 04:30:35 +00:00
firstBlock = chain.blocks[0]
Minor post-merge cleanups (#3945) https://github.com/status-im/nimbus-eth2/pull/3944 The use of nested `awaitWithRetries` calls would have resulted in an unexpected number of retries (3x3). We now use regular `await` in outer layer to avoid the problem. https://github.com/status-im/nimbus-eth2/pull/3943 The new code has an invariant that the `headMerkleizer` field in the `Eth1Chain` is always kept in sync with the blocks stored in the chain. This invariant is now enforced better by doing the necessary merkleizer updates in the `Eth1Chain.addBlock` function, in the `Eth1Chain.init` function and in the `Eth1Chain.reset` function.
2022-08-10 12:31:10 +00:00
depositsInLastPrunedBlock = firstBlock.depositCount -
Fix the corrupted database state on Pyrmont nodes; Add mainnet genesis (#2056) * Handle some web3 timeouts better * Add support for developer .env files * Eth1 improvements; Mainnet genesis state Notable changes: * The deposits table have been removed from the database. The client will no longer process all deposits on start-up. * The network metadata now includes a "state snapshot" of the deposit contract. This allows the client to skip syncing deposits made prior to the snapshot (i.e. genesis). Suitable metadata added for Pyrmont and Mainnet. * The Eth1 monitor won't be started unless there are validators attached to the node. * The genesis detection code is now optional and disabled by default * Bugfix: The client should not produce blocks that will fail validation when it hasn't downloaded the latest deposits yet * Bugfix: Work around the database corruption affecting Pyrmont nodes * Remove metadata for Toledo and Medalla
2020-11-24 21:21:47 +00:00
firstBlock.deposits.lenu64
# advanceMerkleizer should always be called shortly after prunning the chain
doAssert depositsInLastPrunedBlock == merkleizer.getChunkCount
Add Eth1 deposits simulation to block_sim
2020-12-03 04:30:35 +00:00
for blk in chain.blocks:
Fix the corrupted database state on Pyrmont nodes; Add mainnet genesis (#2056) * Handle some web3 timeouts better * Add support for developer .env files * Eth1 improvements; Mainnet genesis state Notable changes: * The deposits table have been removed from the database. The client will no longer process all deposits on start-up. * The network metadata now includes a "state snapshot" of the deposit contract. This allows the client to skip syncing deposits made prior to the snapshot (i.e. genesis). Suitable metadata added for Pyrmont and Mainnet. * The Eth1 monitor won't be started unless there are validators attached to the node. * The genesis detection code is now optional and disabled by default * Bugfix: The client should not produce blocks that will fail validation when it hasn't downloaded the latest deposits yet * Bugfix: Work around the database corruption affecting Pyrmont nodes * Remove metadata for Toledo and Medalla
2020-11-24 21:21:47 +00:00
for deposit in blk.deposits:
if merkleizer.getChunkCount < depositIndex:
merkleizer.addChunk hash_tree_root(deposit).data
else:
return true
return merkleizer.getChunkCount == depositIndex
avoid materializing potentially long deposits seq (#3947) When fetching eth1 data and deposits for a new block proposal, the list of deposits from previous eth1 data to the next one is fully loaded into a `seq`. This can potentially be a very long list in active periods. Changing this to an `iterator` saves memory by ensuring that the entire list is no longer materialized; only the `DepositData` roots are needed.
2022-08-12 13:52:06 +00:00
iterator getDepositsRange*(chain: Eth1Chain, first, last: uint64): DepositData =
Fix the corrupted database state on Pyrmont nodes; Add mainnet genesis (#2056) * Handle some web3 timeouts better * Add support for developer .env files * Eth1 improvements; Mainnet genesis state Notable changes: * The deposits table have been removed from the database. The client will no longer process all deposits on start-up. * The network metadata now includes a "state snapshot" of the deposit contract. This allows the client to skip syncing deposits made prior to the snapshot (i.e. genesis). Suitable metadata added for Pyrmont and Mainnet. * The Eth1 monitor won't be started unless there are validators attached to the node. * The genesis detection code is now optional and disabled by default * Bugfix: The client should not produce blocks that will fail validation when it hasn't downloaded the latest deposits yet * Bugfix: Work around the database corruption affecting Pyrmont nodes * Remove metadata for Toledo and Medalla
2020-11-24 21:21:47 +00:00
# TODO It's possible to make this faster by performing binary search that
# will locate the blocks holding the `first` and `last` indices.
# TODO There is an assumption here that the requested range will be present
avoid materializing potentially long deposits seq (#3947) When fetching eth1 data and deposits for a new block proposal, the list of deposits from previous eth1 data to the next one is fully loaded into a `seq`. This can potentially be a very long list in active periods. Changing this to an `iterator` saves memory by ensuring that the entire list is no longer materialized; only the `DepositData` roots are needed.
2022-08-12 13:52:06 +00:00
# in the Eth1Chain. This should hold true at the call sites right now,
# but we need to guard the pre-conditions better.
Add Eth1 deposits simulation to block_sim
2020-12-03 04:30:35 +00:00
for blk in chain.blocks:
Minor post-merge cleanups (#3945) https://github.com/status-im/nimbus-eth2/pull/3944 The use of nested `awaitWithRetries` calls would have resulted in an unexpected number of retries (3x3). We now use regular `await` in outer layer to avoid the problem. https://github.com/status-im/nimbus-eth2/pull/3943 The new code has an invariant that the `headMerkleizer` field in the `Eth1Chain` is always kept in sync with the blocks stored in the chain. This invariant is now enforced better by doing the necessary merkleizer updates in the `Eth1Chain.addBlock` function, in the `Eth1Chain.init` function and in the `Eth1Chain.reset` function.
2022-08-10 12:31:10 +00:00
if blk.depositCount <= first:
Fix the corrupted database state on Pyrmont nodes; Add mainnet genesis (#2056) * Handle some web3 timeouts better * Add support for developer .env files * Eth1 improvements; Mainnet genesis state Notable changes: * The deposits table have been removed from the database. The client will no longer process all deposits on start-up. * The network metadata now includes a "state snapshot" of the deposit contract. This allows the client to skip syncing deposits made prior to the snapshot (i.e. genesis). Suitable metadata added for Pyrmont and Mainnet. * The Eth1 monitor won't be started unless there are validators attached to the node. * The genesis detection code is now optional and disabled by default * Bugfix: The client should not produce blocks that will fail validation when it hasn't downloaded the latest deposits yet * Bugfix: Work around the database corruption affecting Pyrmont nodes * Remove metadata for Toledo and Medalla
2020-11-24 21:21:47 +00:00
continue
Minor post-merge cleanups (#3945) https://github.com/status-im/nimbus-eth2/pull/3944 The use of nested `awaitWithRetries` calls would have resulted in an unexpected number of retries (3x3). We now use regular `await` in outer layer to avoid the problem. https://github.com/status-im/nimbus-eth2/pull/3943 The new code has an invariant that the `headMerkleizer` field in the `Eth1Chain` is always kept in sync with the blocks stored in the chain. This invariant is now enforced better by doing the necessary merkleizer updates in the `Eth1Chain.addBlock` function, in the `Eth1Chain.init` function and in the `Eth1Chain.reset` function.
2022-08-10 12:31:10 +00:00
let firstDepositIdxInBlk = blk.depositCount - blk.deposits.lenu64
Fix the corrupted database state on Pyrmont nodes; Add mainnet genesis (#2056) * Handle some web3 timeouts better * Add support for developer .env files * Eth1 improvements; Mainnet genesis state Notable changes: * The deposits table have been removed from the database. The client will no longer process all deposits on start-up. * The network metadata now includes a "state snapshot" of the deposit contract. This allows the client to skip syncing deposits made prior to the snapshot (i.e. genesis). Suitable metadata added for Pyrmont and Mainnet. * The Eth1 monitor won't be started unless there are validators attached to the node. * The genesis detection code is now optional and disabled by default * Bugfix: The client should not produce blocks that will fail validation when it hasn't downloaded the latest deposits yet * Bugfix: Work around the database corruption affecting Pyrmont nodes * Remove metadata for Toledo and Medalla
2020-11-24 21:21:47 +00:00
if firstDepositIdxInBlk >= last:
avoid materializing potentially long deposits seq (#3947) When fetching eth1 data and deposits for a new block proposal, the list of deposits from previous eth1 data to the next one is fully loaded into a `seq`. This can potentially be a very long list in active periods. Changing this to an `iterator` saves memory by ensuring that the entire list is no longer materialized; only the `DepositData` roots are needed.
2022-08-12 13:52:06 +00:00
break
Fix the corrupted database state on Pyrmont nodes; Add mainnet genesis (#2056) * Handle some web3 timeouts better * Add support for developer .env files * Eth1 improvements; Mainnet genesis state Notable changes: * The deposits table have been removed from the database. The client will no longer process all deposits on start-up. * The network metadata now includes a "state snapshot" of the deposit contract. This allows the client to skip syncing deposits made prior to the snapshot (i.e. genesis). Suitable metadata added for Pyrmont and Mainnet. * The Eth1 monitor won't be started unless there are validators attached to the node. * The genesis detection code is now optional and disabled by default * Bugfix: The client should not produce blocks that will fail validation when it hasn't downloaded the latest deposits yet * Bugfix: Work around the database corruption affecting Pyrmont nodes * Remove metadata for Toledo and Medalla
2020-11-24 21:21:47 +00:00
for i in 0 ..< blk.deposits.lenu64:
let globalIdx = firstDepositIdxInBlk + i
if globalIdx >= first and globalIdx < last:
avoid materializing potentially long deposits seq (#3947) When fetching eth1 data and deposits for a new block proposal, the list of deposits from previous eth1 data to the next one is fully loaded into a `seq`. This can potentially be a very long list in active periods. Changing this to an `iterator` saves memory by ensuring that the entire list is no longer materialized; only the `DepositData` roots are needed.
2022-08-12 13:52:06 +00:00
yield blk.deposits[i]
Fix the corrupted database state on Pyrmont nodes; Add mainnet genesis (#2056) * Handle some web3 timeouts better * Add support for developer .env files * Eth1 improvements; Mainnet genesis state Notable changes: * The deposits table have been removed from the database. The client will no longer process all deposits on start-up. * The network metadata now includes a "state snapshot" of the deposit contract. This allows the client to skip syncing deposits made prior to the snapshot (i.e. genesis). Suitable metadata added for Pyrmont and Mainnet. * The Eth1 monitor won't be started unless there are validators attached to the node. * The genesis detection code is now optional and disabled by default * Bugfix: The client should not produce blocks that will fail validation when it hasn't downloaded the latest deposits yet * Bugfix: Work around the database corruption affecting Pyrmont nodes * Remove metadata for Toledo and Medalla
2020-11-24 21:21:47 +00:00
implement terminal-total-difficulty-override; keep kintsugi TTDs consistent (#3118) * implement terminal-total-difficulty-override; keep TTD consistent for m2 scripts/docs * use Option[uint64] instead of uint64
2021-11-25 10:53:31 +00:00
func lowerBound(chain: Eth1Chain, depositCount: uint64): Eth1Block =
Less risk for Eth1-induced delays in block proposal
2020-11-30 23:59:35 +00:00
# TODO: This can be replaced with a proper binary search in the
# future, but the `algorithm` module currently requires an
# `openArray`, which the `deques` module can't provide yet.
for eth1Block in chain.blocks:
Minor post-merge cleanups (#3945) https://github.com/status-im/nimbus-eth2/pull/3944 The use of nested `awaitWithRetries` calls would have resulted in an unexpected number of retries (3x3). We now use regular `await` in outer layer to avoid the problem. https://github.com/status-im/nimbus-eth2/pull/3943 The new code has an invariant that the `headMerkleizer` field in the `Eth1Chain` is always kept in sync with the blocks stored in the chain. This invariant is now enforced better by doing the necessary merkleizer updates in the `Eth1Chain.addBlock` function, in the `Eth1Chain.init` function and in the `Eth1Chain.reset` function.
2022-08-10 12:31:10 +00:00
if eth1Block.depositCount > depositCount:
Less risk for Eth1-induced delays in block proposal
2020-11-30 23:59:35 +00:00
return
result = eth1Block
cleanups, partly from kintsugi branch (#3161) * cleanups, partly from kintsugi branch * re-export shortLog(EthBlock) and preserve exception messages in batchVerify and processBatch
2021-12-05 17:32:41 +00:00
proc trackFinalizedState(chain: var Eth1Chain,
finalizedEth1Data: Eth1Data,
Eth1 monitor fixes * Fix a resource leak introduced in https://github.com/status-im/nimbus-eth2/pull/3279 * Don't restart the Eth1 syncing proggress from scratch in case of monitor failures during Eth2 syncing. * Switch to the primary operator as soon as it is back online. * Log the web3 credentials in fewer places Other changes: The 'web3 test' command has been enhanced to obtain and print more data regarding the selected provider.
2022-01-31 17:28:26 +00:00
finalizedStateDepositIndex: uint64,
blockProposalExpected = false): bool =
Support for driving multiple EL nodes from a single Nimbus BN (#4465) * Support for driving multiple EL nodes from a single Nimbus BN Full list of changes: * Eth1Monitor has been renamed to ELManager to match its current responsibilities better. * The ELManager is no longer optional in the code (it won't have a nil value under any circumstances). * The support for subscribing for headers was removed as it only worked with WebSockets and contributed significant complexity while bringing only a very minor advantage. * The `--web3-url` parameter has been deprecated in favor of a new `--el` parameter. The new parameter has a reasonable default value and supports specifying a different JWT for each connection. Each connection can also be configured with a different set of responsibilities (e.g. download deposits, validate blocks and/or produce blocks). On the command-line, these properties can be configured through URL properties stored in the #anchor part of the URL. In TOML files, they come with a very natural syntax (althrough the URL scheme is also supported). * The previously scattered EL-related state and logic is now moved to `eth1_monitor.nim` (this module will be renamed to `el_manager.nim` in a follow-up commit). State is assigned properly either to the `ELManager` or the to individual `ELConnection` objects where appropriate. The ELManager executes all Engine API requests against all attached EL nodes, in parallel. It compares their results and if there is a disagreement regarding the validity of a certain payload, this is detected and the beacon node is protected from publishing a block with a potential execution layer consensus bug in it. The BN provides metrics per EL node for the number of successful or failed requests for each type Engine API requests. If an EL node goes offline and connectivity is resoted later, we report the problem and the remedy in edge-triggered fashion. * More progress towards implementing Deneb block production in the VC and comparing the value of blocks produced by the EL and the builder API. * Adds a Makefile target for the zhejiang testnet
2023-03-05 01:40:21 +00:00
## This function will return true if the ELManager is synced
Minor post-merge cleanups (#3945) https://github.com/status-im/nimbus-eth2/pull/3944 The use of nested `awaitWithRetries` calls would have resulted in an unexpected number of retries (3x3). We now use regular `await` in outer layer to avoid the problem. https://github.com/status-im/nimbus-eth2/pull/3943 The new code has an invariant that the `headMerkleizer` field in the `Eth1Chain` is always kept in sync with the blocks stored in the chain. This invariant is now enforced better by doing the necessary merkleizer updates in the `Eth1Chain.addBlock` function, in the `Eth1Chain.init` function and in the `Eth1Chain.reset` function.
2022-08-10 12:31:10 +00:00
## to the finalization point.
Add Eth1 deposits simulation to block_sim
2020-12-03 04:30:35 +00:00
if chain.blocks.len == 0:
Less risk for Eth1-induced delays in block proposal
2020-11-30 23:59:35 +00:00
debug "Eth1 chain not initialized"
return false
Add Eth1 deposits simulation to block_sim
2020-12-03 04:30:35 +00:00
let latest = chain.blocks.peekLast
Minor post-merge cleanups (#3945) https://github.com/status-im/nimbus-eth2/pull/3944 The use of nested `awaitWithRetries` calls would have resulted in an unexpected number of retries (3x3). We now use regular `await` in outer layer to avoid the problem. https://github.com/status-im/nimbus-eth2/pull/3943 The new code has an invariant that the `headMerkleizer` field in the `Eth1Chain` is always kept in sync with the blocks stored in the chain. This invariant is now enforced better by doing the necessary merkleizer updates in the `Eth1Chain.addBlock` function, in the `Eth1Chain.init` function and in the `Eth1Chain.reset` function.
2022-08-10 12:31:10 +00:00
if latest.depositCount < finalizedEth1Data.deposit_count:
Eth1 monitor fixes * Fix a resource leak introduced in https://github.com/status-im/nimbus-eth2/pull/3279 * Don't restart the Eth1 syncing proggress from scratch in case of monitor failures during Eth2 syncing. * Switch to the primary operator as soon as it is back online. * Log the web3 credentials in fewer places Other changes: The 'web3 test' command has been enhanced to obtain and print more data regarding the selected provider.
2022-01-31 17:28:26 +00:00
if blockProposalExpected:
error "The Eth1 chain is not synced",
Minor post-merge cleanups (#3945) https://github.com/status-im/nimbus-eth2/pull/3944 The use of nested `awaitWithRetries` calls would have resulted in an unexpected number of retries (3x3). We now use regular `await` in outer layer to avoid the problem. https://github.com/status-im/nimbus-eth2/pull/3943 The new code has an invariant that the `headMerkleizer` field in the `Eth1Chain` is always kept in sync with the blocks stored in the chain. This invariant is now enforced better by doing the necessary merkleizer updates in the `Eth1Chain.addBlock` function, in the `Eth1Chain.init` function and in the `Eth1Chain.reset` function.
2022-08-10 12:31:10 +00:00
ourDepositsCount = latest.depositCount,
Eth1 monitor fixes * Fix a resource leak introduced in https://github.com/status-im/nimbus-eth2/pull/3279 * Don't restart the Eth1 syncing proggress from scratch in case of monitor failures during Eth2 syncing. * Switch to the primary operator as soon as it is back online. * Log the web3 credentials in fewer places Other changes: The 'web3 test' command has been enhanced to obtain and print more data regarding the selected provider.
2022-01-31 17:28:26 +00:00
targetDepositsCount = finalizedEth1Data.deposit_count
Less risk for Eth1-induced delays in block proposal
2020-11-30 23:59:35 +00:00
return false
Add Eth1 deposits simulation to block_sim
2020-12-03 04:30:35 +00:00
let matchingBlock = chain.lowerBound(finalizedEth1Data.deposit_count)
Less risk for Eth1-induced delays in block proposal
2020-11-30 23:59:35 +00:00
result = if matchingBlock != nil:
Minor post-merge cleanups (#3945) https://github.com/status-im/nimbus-eth2/pull/3944 The use of nested `awaitWithRetries` calls would have resulted in an unexpected number of retries (3x3). We now use regular `await` in outer layer to avoid the problem. https://github.com/status-im/nimbus-eth2/pull/3943 The new code has an invariant that the `headMerkleizer` field in the `Eth1Chain` is always kept in sync with the blocks stored in the chain. This invariant is now enforced better by doing the necessary merkleizer updates in the `Eth1Chain.addBlock` function, in the `Eth1Chain.init` function and in the `Eth1Chain.reset` function.
2022-08-10 12:31:10 +00:00
if matchingBlock.depositRoot == finalizedEth1Data.deposit_root:
Fix the corrupted database state on Pyrmont nodes; Add mainnet genesis (#2056) * Handle some web3 timeouts better * Add support for developer .env files * Eth1 improvements; Mainnet genesis state Notable changes: * The deposits table have been removed from the database. The client will no longer process all deposits on start-up. * The network metadata now includes a "state snapshot" of the deposit contract. This allows the client to skip syncing deposits made prior to the snapshot (i.e. genesis). Suitable metadata added for Pyrmont and Mainnet. * The Eth1 monitor won't be started unless there are validators attached to the node. * The genesis detection code is now optional and disabled by default * Bugfix: The client should not produce blocks that will fail validation when it hasn't downloaded the latest deposits yet * Bugfix: Work around the database corruption affecting Pyrmont nodes * Remove metadata for Toledo and Medalla
2020-11-24 21:21:47 +00:00
true
else:
error "Corrupted deposits history detected",
Minor post-merge cleanups (#3945) https://github.com/status-im/nimbus-eth2/pull/3944 The use of nested `awaitWithRetries` calls would have resulted in an unexpected number of retries (3x3). We now use regular `await` in outer layer to avoid the problem. https://github.com/status-im/nimbus-eth2/pull/3943 The new code has an invariant that the `headMerkleizer` field in the `Eth1Chain` is always kept in sync with the blocks stored in the chain. This invariant is now enforced better by doing the necessary merkleizer updates in the `Eth1Chain.addBlock` function, in the `Eth1Chain.init` function and in the `Eth1Chain.reset` function.
2022-08-10 12:31:10 +00:00
ourDepositsCount = matchingBlock.depositCount,
fix misspelling in log message (#5069)
2023-06-13 17:08:41 +00:00
targetDepositsCount = finalizedEth1Data.deposit_count,
Minor post-merge cleanups (#3945) https://github.com/status-im/nimbus-eth2/pull/3944 The use of nested `awaitWithRetries` calls would have resulted in an unexpected number of retries (3x3). We now use regular `await` in outer layer to avoid the problem. https://github.com/status-im/nimbus-eth2/pull/3943 The new code has an invariant that the `headMerkleizer` field in the `Eth1Chain` is always kept in sync with the blocks stored in the chain. This invariant is now enforced better by doing the necessary merkleizer updates in the `Eth1Chain.addBlock` function, in the `Eth1Chain.init` function and in the `Eth1Chain.reset` function.
2022-08-10 12:31:10 +00:00
ourDepositsRoot = matchingBlock.depositRoot,
Add Eth1 deposits simulation to block_sim
2020-12-03 04:30:35 +00:00
targetDepositsRoot = finalizedEth1Data.deposit_root
Fix #2261 Also bumps Confutils to allow setting the hidden --web3-mode param (to allow testing the eth1 syncing without validators)
2021-01-29 21:21:44 +00:00
chain.hasConsensusViolation = true
Hotfix for a failing assert during block production
2020-11-04 09:06:58 +00:00
false
else:
Less risk for Eth1-induced delays in block proposal
2020-11-30 23:59:35 +00:00
error "The Eth1 chain is in inconsistent state",
Fix the corrupted database state on Pyrmont nodes; Add mainnet genesis (#2056) * Handle some web3 timeouts better * Add support for developer .env files * Eth1 improvements; Mainnet genesis state Notable changes: * The deposits table have been removed from the database. The client will no longer process all deposits on start-up. * The network metadata now includes a "state snapshot" of the deposit contract. This allows the client to skip syncing deposits made prior to the snapshot (i.e. genesis). Suitable metadata added for Pyrmont and Mainnet. * The Eth1 monitor won't be started unless there are validators attached to the node. * The genesis detection code is now optional and disabled by default * Bugfix: The client should not produce blocks that will fail validation when it hasn't downloaded the latest deposits yet * Bugfix: Work around the database corruption affecting Pyrmont nodes * Remove metadata for Toledo and Medalla
2020-11-24 21:21:47 +00:00
checkpointHash = finalizedEth1Data.block_hash,
checkpointDeposits = finalizedEth1Data.deposit_count,
Add Eth1 deposits simulation to block_sim
2020-12-03 04:30:35 +00:00
localChainStart = shortLog(chain.blocks.peekFirst),
localChainEnd = shortLog(chain.blocks.peekLast)
Fix #2261 Also bumps Confutils to allow setting the hidden --web3-mode param (to allow testing the eth1 syncing without validators)
2021-01-29 21:21:44 +00:00
chain.hasConsensusViolation = true
Hotfix for a failing assert during block production
2020-11-04 09:06:58 +00:00
false
Don't require requests that might fail on non-archive Geth nodes
2020-10-26 08:55:10 +00:00
Less risk for Eth1-induced delays in block proposal
2020-11-30 23:59:35 +00:00
if result:
Add Eth1 deposits simulation to block_sim
2020-12-03 04:30:35 +00:00
chain.pruneOldBlocks(finalizedStateDepositIndex)
Support for driving multiple EL nodes from a single Nimbus BN (#4465) * Support for driving multiple EL nodes from a single Nimbus BN Full list of changes: * Eth1Monitor has been renamed to ELManager to match its current responsibilities better. * The ELManager is no longer optional in the code (it won't have a nil value under any circumstances). * The support for subscribing for headers was removed as it only worked with WebSockets and contributed significant complexity while bringing only a very minor advantage. * The `--web3-url` parameter has been deprecated in favor of a new `--el` parameter. The new parameter has a reasonable default value and supports specifying a different JWT for each connection. Each connection can also be configured with a different set of responsibilities (e.g. download deposits, validate blocks and/or produce blocks). On the command-line, these properties can be configured through URL properties stored in the #anchor part of the URL. In TOML files, they come with a very natural syntax (althrough the URL scheme is also supported). * The previously scattered EL-related state and logic is now moved to `eth1_monitor.nim` (this module will be renamed to `el_manager.nim` in a follow-up commit). State is assigned properly either to the `ELManager` or the to individual `ELConnection` objects where appropriate. The ELManager executes all Engine API requests against all attached EL nodes, in parallel. It compares their results and if there is a disagreement regarding the validity of a certain payload, this is detected and the beacon node is protected from publishing a block with a potential execution layer consensus bug in it. The BN provides metrics per EL node for the number of successful or failed requests for each type Engine API requests. If an EL node goes offline and connectivity is resoted later, we report the problem and the remedy in edge-triggered fashion. * More progress towards implementing Deneb block production in the VC and comparing the value of blocks produced by the EL and the builder API. * Adds a Makefile target for the zhejiang testnet
2023-03-05 01:40:21 +00:00
template trackFinalizedState*(m: ELManager,
Add Eth1 deposits simulation to block_sim
2020-12-03 04:30:35 +00:00
finalizedEth1Data: Eth1Data,
finalizedStateDepositIndex: uint64): bool =
Support for driving multiple EL nodes from a single Nimbus BN (#4465) * Support for driving multiple EL nodes from a single Nimbus BN Full list of changes: * Eth1Monitor has been renamed to ELManager to match its current responsibilities better. * The ELManager is no longer optional in the code (it won't have a nil value under any circumstances). * The support for subscribing for headers was removed as it only worked with WebSockets and contributed significant complexity while bringing only a very minor advantage. * The `--web3-url` parameter has been deprecated in favor of a new `--el` parameter. The new parameter has a reasonable default value and supports specifying a different JWT for each connection. Each connection can also be configured with a different set of responsibilities (e.g. download deposits, validate blocks and/or produce blocks). On the command-line, these properties can be configured through URL properties stored in the #anchor part of the URL. In TOML files, they come with a very natural syntax (althrough the URL scheme is also supported). * The previously scattered EL-related state and logic is now moved to `eth1_monitor.nim` (this module will be renamed to `el_manager.nim` in a follow-up commit). State is assigned properly either to the `ELManager` or the to individual `ELConnection` objects where appropriate. The ELManager executes all Engine API requests against all attached EL nodes, in parallel. It compares their results and if there is a disagreement regarding the validity of a certain payload, this is detected and the beacon node is protected from publishing a block with a potential execution layer consensus bug in it. The BN provides metrics per EL node for the number of successful or failed requests for each type Engine API requests. If an EL node goes offline and connectivity is resoted later, we report the problem and the remedy in edge-triggered fashion. * More progress towards implementing Deneb block production in the VC and comparing the value of blocks produced by the EL and the builder API. * Adds a Makefile target for the zhejiang testnet
2023-03-05 01:40:21 +00:00
trackFinalizedState(m.eth1Chain, finalizedEth1Data, finalizedStateDepositIndex)
Fix the corrupted database state on Pyrmont nodes; Add mainnet genesis (#2056) * Handle some web3 timeouts better * Add support for developer .env files * Eth1 improvements; Mainnet genesis state Notable changes: * The deposits table have been removed from the database. The client will no longer process all deposits on start-up. * The network metadata now includes a "state snapshot" of the deposit contract. This allows the client to skip syncing deposits made prior to the snapshot (i.e. genesis). Suitable metadata added for Pyrmont and Mainnet. * The Eth1 monitor won't be started unless there are validators attached to the node. * The genesis detection code is now optional and disabled by default * Bugfix: The client should not produce blocks that will fail validation when it hasn't downloaded the latest deposits yet * Bugfix: Work around the database corruption affecting Pyrmont nodes * Remove metadata for Toledo and Medalla
2020-11-24 21:21:47 +00:00
automated consensus spec URL updating to v1.4.0-beta.3 (#5514)
2023-10-19 10:26:38 +00:00
# https://github.com/ethereum/consensus-specs/blob/v1.4.0-beta.3/specs/phase0/validator.md#get_eth1_data
Add Eth1 deposits simulation to block_sim
2020-12-03 04:30:35 +00:00
proc getBlockProposalData*(chain: var Eth1Chain,
use ForkedHashedBeaconState in StateData (#2634) * use ForkedHashedBeaconState in StateData * fix FAR_FUTURE_EPOCH -> slot overflow; almost always use assign() * avoid stack allocation in maybeUpgradeStateToAltair() * create and use dispatch functions for check_attester_slashing(), check_proposer_slashing(), and check_voluntary_exit() * use getStateRoot() instead of various state.data.hbsPhase0.root * remove withStateVars.hashedState(), which doesn't work as a design anymore * introduce spec/datatypes/altair into beacon_chain_db * fix inefficient codegen for getStateField(largeStateField) * state_transition_slots() doesn't either need/use blocks or runtime presets * combine process_slots(HBS)/state_transition_slots(HBS) which differ only in last-slot htr optimization * getStateField(StateData, ...) was replaced by getStateField(ForkedHashedBeaconState, ...) * fix rollback * switch some state_transition(), process_slots, makeTestBlocks(), etc to use ForkedHashedBeaconState * remove state_transition(phase0.HashedBeaconState) * remove process_slots(phase0.HashedBeaconState) * remove state_transition_block(phase0.HashedBeaconState) * remove unused callWithBS(); separate case expression from if statement * switch back from nested-ref-object construction to (ref Foo)(Bar())
2021-06-11 17:51:46 +00:00
state: ForkedHashedBeaconState,
Less risk for Eth1-induced delays in block proposal
2020-11-30 23:59:35 +00:00
finalizedEth1Data: Eth1Data,
finalizedStateDepositIndex: uint64): BlockProposalEth1Data =
let
periodStart = voting_period_start_time(state)
Add Eth1 deposits simulation to block_sim
2020-12-03 04:30:35 +00:00
hasLatestDeposits = chain.trackFinalizedState(finalizedEth1Data,
Eth1 monitor fixes * Fix a resource leak introduced in https://github.com/status-im/nimbus-eth2/pull/3279 * Don't restart the Eth1 syncing proggress from scratch in case of monitor failures during Eth2 syncing. * Switch to the primary operator as soon as it is back online. * Log the web3 credentials in fewer places Other changes: The 'web3 test' command has been enhanced to obtain and print more data regarding the selected provider.
2022-01-31 17:28:26 +00:00
finalizedStateDepositIndex,
blockProposalExpected = true)
Spec-compliant implementation of Eth1 monitoring; Eth1-enabled local sim BEWARE! This commit will trigger a stack overflow during local sim
2020-03-24 11:13:07 +00:00
Tentative fix for the Pyrmont Eth1 voting issue
2020-11-19 17:19:03 +00:00
var otherVotesCountTable = initCountTable[Eth1Data]()
use StateData in place of BeaconState outside state transition code (#2551) * use StateData in place of BeaconState outside state transition code * propagate more StateData usage * remove withStateVars().state * wrap get_beacon_committee(BeaconState, ...) as gbc(StateData, ...) * switch makeAttestation() to use StateData * use StateData wrapper/dispatcher for get_committee_count_per_slot() * convert AttestationCache.init(), weak subjectivity functions, and updateValidatorMetrics() * add get_shuffled_active_validator_indices(StateData) and get_block_root_at_slot(StateData) * switch makeAttestationData() to StateData * sync AllTests-mainnet.md after rebase
2021-05-21 09:23:28 +00:00
for vote in getStateField(state, eth1_data_votes):
Add Eth1 deposits simulation to block_sim
2020-12-03 04:30:35 +00:00
let eth1Block = chain.findBlock(vote)
Stronger validation for third-party Eth1Data votes
2021-04-30 23:00:03 +00:00
if eth1Block != nil and
Minor post-merge cleanups (#3945) https://github.com/status-im/nimbus-eth2/pull/3944 The use of nested `awaitWithRetries` calls would have resulted in an unexpected number of retries (3x3). We now use regular `await` in outer layer to avoid the problem. https://github.com/status-im/nimbus-eth2/pull/3943 The new code has an invariant that the `headMerkleizer` field in the `Eth1Chain` is always kept in sync with the blocks stored in the chain. This invariant is now enforced better by doing the necessary merkleizer updates in the `Eth1Chain.addBlock` function, in the `Eth1Chain.init` function and in the `Eth1Chain.reset` function.
2022-08-10 12:31:10 +00:00
eth1Block.depositRoot == vote.deposit_root and
use StateData in place of BeaconState outside state transition code (#2551) * use StateData in place of BeaconState outside state transition code * propagate more StateData usage * remove withStateVars().state * wrap get_beacon_committee(BeaconState, ...) as gbc(StateData, ...) * switch makeAttestation() to use StateData * use StateData wrapper/dispatcher for get_committee_count_per_slot() * convert AttestationCache.init(), weak subjectivity functions, and updateValidatorMetrics() * add get_shuffled_active_validator_indices(StateData) and get_block_root_at_slot(StateData) * switch makeAttestationData() to StateData * sync AllTests-mainnet.md after rebase
2021-05-21 09:23:28 +00:00
vote.deposit_count >= getStateField(state, eth1_data).deposit_count and
Implement split preset/config support (#2710) * Implement split preset/config support This is the initial bulk refactor to introduce runtime config values in a number of places, somewhat replacing the existing mechanism of loading network metadata. It still needs more work, this is the initial refactor that introduces runtime configuration in some of the places that need it. The PR changes the way presets and constants work, to match the spec. In particular, a "preset" now refers to the compile-time configuration while a "cfg" or "RuntimeConfig" is the dynamic part. A single binary can support either mainnet or minimal, but not both. Support for other presets has been removed completely (can be readded, in case there's need). There's a number of outstanding tasks: * `SECONDS_PER_SLOT` still needs fixing * loading custom runtime configs needs redoing * checking constants against YAML file * yeerongpilly support `build/nimbus_beacon_node --network=yeerongpilly --discv5:no --log-level=DEBUG` * load fork epoch from config * fix fork digest sent in status * nicer error string for request failures * fix tools * one more * fixup * fixup * fixup * use "standard" network definition folder in local testnet Files are loaded from their standard locations, including genesis etc, to conform to the format used in the `eth2-networks` repo. * fix launch scripts, allow unknown config values * fix base config of rest test * cleanups * bundle mainnet config using common loader * fix spec links and names * only include supported preset in binary * drop yeerongpilly, add altair-devnet-0, support boot_enr.yaml
2021-07-12 13:01:38 +00:00
is_candidate_block(chain.cfg, eth1Block, periodStart):
Tentative fix for the Pyrmont Eth1 voting issue
2020-11-19 17:19:03 +00:00
otherVotesCountTable.inc vote
Add more eth1-related logging
2020-11-20 14:05:37 +00:00
else:
Stronger validation for third-party Eth1Data votes
2021-04-30 23:00:03 +00:00
debug "Ignoring eth1 vote",
root = vote.block_hash,
deposits = vote.deposit_count,
depositsRoot = vote.deposit_root,
use StateData in place of BeaconState outside state transition code (#2551) * use StateData in place of BeaconState outside state transition code * propagate more StateData usage * remove withStateVars().state * wrap get_beacon_committee(BeaconState, ...) as gbc(StateData, ...) * switch makeAttestation() to use StateData * use StateData wrapper/dispatcher for get_committee_count_per_slot() * convert AttestationCache.init(), weak subjectivity functions, and updateValidatorMetrics() * add get_shuffled_active_validator_indices(StateData) and get_block_root_at_slot(StateData) * switch makeAttestationData() to StateData * sync AllTests-mainnet.md after rebase
2021-05-21 09:23:28 +00:00
localDeposits = getStateField(state, eth1_data).deposit_count
Spec-compliant implementation of Eth1 monitoring; Eth1-enabled local sim BEWARE! This commit will trigger a stack overflow during local sim
2020-03-24 11:13:07 +00:00
Fix invalid merkle proof generation when there is a large number of unprocessed deposits (#3607)
2022-04-27 09:39:41 +00:00
let
stateDepositIdx = getStateField(state, eth1_deposit_index)
stateDepositsCount = getStateField(state, eth1_data).deposit_count
# A valid state should never have this condition, but it doesn't hurt
# to be extra defensive here because we are working with uint types
var pendingDepositsCount = if stateDepositsCount > stateDepositIdx:
stateDepositsCount - stateDepositIdx
else:
0
Spec-compliant implementation of Eth1 monitoring; Eth1-enabled local sim BEWARE! This commit will trigger a stack overflow during local sim
2020-03-24 11:13:07 +00:00
if otherVotesCountTable.len > 0:
Tentative fix for the Pyrmont Eth1 voting issue
2020-11-19 17:19:03 +00:00
let (winningVote, votes) = otherVotesCountTable.largest
Add more eth1-related logging
2020-11-20 14:05:37 +00:00
debug "Voting on eth1 head with majority", votes
Tentative fix for the Pyrmont Eth1 voting issue
2020-11-19 17:19:03 +00:00
result.vote = winningVote
Address #1584 Don't keep all deposits in memory (persist them to disk)
2020-10-12 01:07:20 +00:00
if uint64((votes + 1) * 2) > SLOTS_PER_ETH1_VOTING_PERIOD:
Fix invalid merkle proof generation when there is a large number of unprocessed deposits (#3607)
2022-04-27 09:39:41 +00:00
pendingDepositsCount = winningVote.deposit_count - stateDepositIdx
Spec-compliant implementation of Eth1 monitoring; Eth1-enabled local sim BEWARE! This commit will trigger a stack overflow during local sim
2020-03-24 11:13:07 +00:00
else:
Add Eth1 deposits simulation to block_sim
2020-12-03 04:30:35 +00:00
let latestBlock = chain.latestCandidateBlock(periodStart)
Address #1584 Don't keep all deposits in memory (persist them to disk)
2020-10-12 01:07:20 +00:00
if latestBlock == nil:
Add more eth1-related logging
2020-11-20 14:05:37 +00:00
debug "No acceptable eth1 votes and no recent candidates. Voting no change"
use StateData in place of BeaconState outside state transition code (#2551) * use StateData in place of BeaconState outside state transition code * propagate more StateData usage * remove withStateVars().state * wrap get_beacon_committee(BeaconState, ...) as gbc(StateData, ...) * switch makeAttestation() to use StateData * use StateData wrapper/dispatcher for get_committee_count_per_slot() * convert AttestationCache.init(), weak subjectivity functions, and updateValidatorMetrics() * add get_shuffled_active_validator_indices(StateData) and get_block_root_at_slot(StateData) * switch makeAttestationData() to StateData * sync AllTests-mainnet.md after rebase
2021-05-21 09:23:28 +00:00
result.vote = getStateField(state, eth1_data)
Address #1584 Don't keep all deposits in memory (persist them to disk)
2020-10-12 01:07:20 +00:00
else:
Add more eth1-related logging
2020-11-20 14:05:37 +00:00
debug "No acceptable eth1 votes. Voting for latest candidate"
Minor post-merge cleanups (#3945) https://github.com/status-im/nimbus-eth2/pull/3944 The use of nested `awaitWithRetries` calls would have resulted in an unexpected number of retries (3x3). We now use regular `await` in outer layer to avoid the problem. https://github.com/status-im/nimbus-eth2/pull/3943 The new code has an invariant that the `headMerkleizer` field in the `Eth1Chain` is always kept in sync with the blocks stored in the chain. This invariant is now enforced better by doing the necessary merkleizer updates in the `Eth1Chain.addBlock` function, in the `Eth1Chain.init` function and in the `Eth1Chain.reset` function.
2022-08-10 12:31:10 +00:00
result.vote = latestBlock.toVoteData
Update get_eth1_data from v0.11.1 to v1.0.0-rc
2020-10-15 17:30:33 +00:00
Fix the corrupted database state on Pyrmont nodes; Add mainnet genesis (#2056) * Handle some web3 timeouts better * Add support for developer .env files * Eth1 improvements; Mainnet genesis state Notable changes: * The deposits table have been removed from the database. The client will no longer process all deposits on start-up. * The network metadata now includes a "state snapshot" of the deposit contract. This allows the client to skip syncing deposits made prior to the snapshot (i.e. genesis). Suitable metadata added for Pyrmont and Mainnet. * The Eth1 monitor won't be started unless there are validators attached to the node. * The genesis detection code is now optional and disabled by default * Bugfix: The client should not produce blocks that will fail validation when it hasn't downloaded the latest deposits yet * Bugfix: Work around the database corruption affecting Pyrmont nodes * Remove metadata for Toledo and Medalla
2020-11-24 21:21:47 +00:00
if pendingDepositsCount > 0:
if hasLatestDeposits:
Fix a block proposal issue caused by incorrect merkle proofs The key change here is that `addChunksAndGenMerkleProofs` is called with all pending deposits instead of just the deposits included in the block. The later was effectively producing merkle proofs against a different root.
2020-12-02 18:15:36 +00:00
let
totalDepositsInNewBlock = min(MAX_DEPOSITS, pendingDepositsCount)
Fix invalid merkle proof generation when there is a large number of unprocessed deposits (#3607)
2022-04-27 09:39:41 +00:00
postStateDepositIdx = stateDepositIdx + pendingDepositsCount
avoid materializing potentially long deposits seq (#3947) When fetching eth1 data and deposits for a new block proposal, the list of deposits from previous eth1 data to the next one is fully loaded into a `seq`. This can potentially be a very long list in active periods. Changing this to an `iterator` saves memory by ensuring that the entire list is no longer materialized; only the `DepositData` roots are needed.
2022-08-12 13:52:06 +00:00
var
deposits = newSeqOfCap[DepositData](totalDepositsInNewBlock)
depositRoots = newSeqOfCap[Eth2Digest](pendingDepositsCount)
for data in chain.getDepositsRange(stateDepositIdx, postStateDepositIdx):
if deposits.lenu64 < totalDepositsInNewBlock:
deposits.add data
depositRoots.add hash_tree_root(data)
Fix the corrupted database state on Pyrmont nodes; Add mainnet genesis (#2056) * Handle some web3 timeouts better * Add support for developer .env files * Eth1 improvements; Mainnet genesis state Notable changes: * The deposits table have been removed from the database. The client will no longer process all deposits on start-up. * The network metadata now includes a "state snapshot" of the deposit contract. This allows the client to skip syncing deposits made prior to the snapshot (i.e. genesis). Suitable metadata added for Pyrmont and Mainnet. * The Eth1 monitor won't be started unless there are validators attached to the node. * The genesis detection code is now optional and disabled by default * Bugfix: The client should not produce blocks that will fail validation when it hasn't downloaded the latest deposits yet * Bugfix: Work around the database corruption affecting Pyrmont nodes * Remove metadata for Toledo and Medalla
2020-11-24 21:21:47 +00:00
ssz: bump (#5463) more `zeroMem` avoidance, value semantics for merkleizer
2023-09-28 11:59:33 +00:00
var scratchMerkleizer = chain.finalizedDepositsMerkleizer
Fix invalid merkle proof generation when there is a large number of unprocessed deposits (#3607)
2022-04-27 09:39:41 +00:00
if chain.advanceMerkleizer(scratchMerkleizer, stateDepositIdx):
Fix the corrupted database state on Pyrmont nodes; Add mainnet genesis (#2056) * Handle some web3 timeouts better * Add support for developer .env files * Eth1 improvements; Mainnet genesis state Notable changes: * The deposits table have been removed from the database. The client will no longer process all deposits on start-up. * The network metadata now includes a "state snapshot" of the deposit contract. This allows the client to skip syncing deposits made prior to the snapshot (i.e. genesis). Suitable metadata added for Pyrmont and Mainnet. * The Eth1 monitor won't be started unless there are validators attached to the node. * The genesis detection code is now optional and disabled by default * Bugfix: The client should not produce blocks that will fail validation when it hasn't downloaded the latest deposits yet * Bugfix: Work around the database corruption affecting Pyrmont nodes * Remove metadata for Toledo and Medalla
2020-11-24 21:21:47 +00:00
let proofs = scratchMerkleizer.addChunksAndGenMerkleProofs(depositRoots)
Fix a block proposal issue caused by incorrect merkle proofs The key change here is that `addChunksAndGenMerkleProofs` is called with all pending deposits instead of just the deposits included in the block. The later was effectively producing merkle proofs against a different root.
2020-12-02 18:15:36 +00:00
for i in 0 ..< totalDepositsInNewBlock:
var proof: array[33, Eth2Digest]
proof[0..31] = proofs.getProof(i.int)
proof[32] = default(Eth2Digest)
Fix invalid merkle proof generation when there is a large number of unprocessed deposits (#3607)
2022-04-27 09:39:41 +00:00
proof[32].data[0..7] = toBytesLE uint64(postStateDepositIdx)
Fix a block proposal issue caused by incorrect merkle proofs The key change here is that `addChunksAndGenMerkleProofs` is called with all pending deposits instead of just the deposits included in the block. The later was effectively producing merkle proofs against a different root.
2020-12-02 18:15:36 +00:00
result.deposits.add Deposit(data: deposits[i], proof: proof)
Fix the corrupted database state on Pyrmont nodes; Add mainnet genesis (#2056) * Handle some web3 timeouts better * Add support for developer .env files * Eth1 improvements; Mainnet genesis state Notable changes: * The deposits table have been removed from the database. The client will no longer process all deposits on start-up. * The network metadata now includes a "state snapshot" of the deposit contract. This allows the client to skip syncing deposits made prior to the snapshot (i.e. genesis). Suitable metadata added for Pyrmont and Mainnet. * The Eth1 monitor won't be started unless there are validators attached to the node. * The genesis detection code is now optional and disabled by default * Bugfix: The client should not produce blocks that will fail validation when it hasn't downloaded the latest deposits yet * Bugfix: Work around the database corruption affecting Pyrmont nodes * Remove metadata for Toledo and Medalla
2020-11-24 21:21:47 +00:00
else:
error "The Eth1 chain is in inconsistent state" # This should not really happen
result.hasMissingDeposits = true
else:
result.hasMissingDeposits = true
Spec-compliant implementation of Eth1 monitoring; Eth1-enabled local sim BEWARE! This commit will trigger a stack overflow during local sim
2020-03-24 11:13:07 +00:00
Support for driving multiple EL nodes from a single Nimbus BN (#4465) * Support for driving multiple EL nodes from a single Nimbus BN Full list of changes: * Eth1Monitor has been renamed to ELManager to match its current responsibilities better. * The ELManager is no longer optional in the code (it won't have a nil value under any circumstances). * The support for subscribing for headers was removed as it only worked with WebSockets and contributed significant complexity while bringing only a very minor advantage. * The `--web3-url` parameter has been deprecated in favor of a new `--el` parameter. The new parameter has a reasonable default value and supports specifying a different JWT for each connection. Each connection can also be configured with a different set of responsibilities (e.g. download deposits, validate blocks and/or produce blocks). On the command-line, these properties can be configured through URL properties stored in the #anchor part of the URL. In TOML files, they come with a very natural syntax (althrough the URL scheme is also supported). * The previously scattered EL-related state and logic is now moved to `eth1_monitor.nim` (this module will be renamed to `el_manager.nim` in a follow-up commit). State is assigned properly either to the `ELManager` or the to individual `ELConnection` objects where appropriate. The ELManager executes all Engine API requests against all attached EL nodes, in parallel. It compares their results and if there is a disagreement regarding the validity of a certain payload, this is detected and the beacon node is protected from publishing a block with a potential execution layer consensus bug in it. The BN provides metrics per EL node for the number of successful or failed requests for each type Engine API requests. If an EL node goes offline and connectivity is resoted later, we report the problem and the remedy in edge-triggered fashion. * More progress towards implementing Deneb block production in the VC and comparing the value of blocks produced by the EL and the builder API. * Adds a Makefile target for the zhejiang testnet
2023-03-05 01:40:21 +00:00
template getBlockProposalData*(m: ELManager,
use ForkedHashedBeaconState in StateData (#2634) * use ForkedHashedBeaconState in StateData * fix FAR_FUTURE_EPOCH -> slot overflow; almost always use assign() * avoid stack allocation in maybeUpgradeStateToAltair() * create and use dispatch functions for check_attester_slashing(), check_proposer_slashing(), and check_voluntary_exit() * use getStateRoot() instead of various state.data.hbsPhase0.root * remove withStateVars.hashedState(), which doesn't work as a design anymore * introduce spec/datatypes/altair into beacon_chain_db * fix inefficient codegen for getStateField(largeStateField) * state_transition_slots() doesn't either need/use blocks or runtime presets * combine process_slots(HBS)/state_transition_slots(HBS) which differ only in last-slot htr optimization * getStateField(StateData, ...) was replaced by getStateField(ForkedHashedBeaconState, ...) * fix rollback * switch some state_transition(), process_slots, makeTestBlocks(), etc to use ForkedHashedBeaconState * remove state_transition(phase0.HashedBeaconState) * remove process_slots(phase0.HashedBeaconState) * remove state_transition_block(phase0.HashedBeaconState) * remove unused callWithBS(); separate case expression from if statement * switch back from nested-ref-object construction to (ref Foo)(Bar())
2021-06-11 17:51:46 +00:00
state: ForkedHashedBeaconState,
Add Eth1 deposits simulation to block_sim
2020-12-03 04:30:35 +00:00
finalizedEth1Data: Eth1Data,
JWT support (#3561)
2022-03-31 14:43:05 +00:00
finalizedStateDepositIndex: uint64):
BlockProposalEth1Data =
getBlockProposalData(
Support for driving multiple EL nodes from a single Nimbus BN (#4465) * Support for driving multiple EL nodes from a single Nimbus BN Full list of changes: * Eth1Monitor has been renamed to ELManager to match its current responsibilities better. * The ELManager is no longer optional in the code (it won't have a nil value under any circumstances). * The support for subscribing for headers was removed as it only worked with WebSockets and contributed significant complexity while bringing only a very minor advantage. * The `--web3-url` parameter has been deprecated in favor of a new `--el` parameter. The new parameter has a reasonable default value and supports specifying a different JWT for each connection. Each connection can also be configured with a different set of responsibilities (e.g. download deposits, validate blocks and/or produce blocks). On the command-line, these properties can be configured through URL properties stored in the #anchor part of the URL. In TOML files, they come with a very natural syntax (althrough the URL scheme is also supported). * The previously scattered EL-related state and logic is now moved to `eth1_monitor.nim` (this module will be renamed to `el_manager.nim` in a follow-up commit). State is assigned properly either to the `ELManager` or the to individual `ELConnection` objects where appropriate. The ELManager executes all Engine API requests against all attached EL nodes, in parallel. It compares their results and if there is a disagreement regarding the validity of a certain payload, this is detected and the beacon node is protected from publishing a block with a potential execution layer consensus bug in it. The BN provides metrics per EL node for the number of successful or failed requests for each type Engine API requests. If an EL node goes offline and connectivity is resoted later, we report the problem and the remedy in edge-triggered fashion. * More progress towards implementing Deneb block production in the VC and comparing the value of blocks produced by the EL and the builder API. * Adds a Makefile target for the zhejiang testnet
2023-03-05 01:40:21 +00:00
m.eth1Chain, state, finalizedEth1Data, finalizedStateDepositIndex)
JWT support (#3561)
2022-03-31 14:43:05 +00:00
simplify ELMonitor fcU payload attributes handling (#4696)
2023-03-06 16:19:15 +00:00
func new*(T: type ELConnection,
Support for driving multiple EL nodes from a single Nimbus BN (#4465) * Support for driving multiple EL nodes from a single Nimbus BN Full list of changes: * Eth1Monitor has been renamed to ELManager to match its current responsibilities better. * The ELManager is no longer optional in the code (it won't have a nil value under any circumstances). * The support for subscribing for headers was removed as it only worked with WebSockets and contributed significant complexity while bringing only a very minor advantage. * The `--web3-url` parameter has been deprecated in favor of a new `--el` parameter. The new parameter has a reasonable default value and supports specifying a different JWT for each connection. Each connection can also be configured with a different set of responsibilities (e.g. download deposits, validate blocks and/or produce blocks). On the command-line, these properties can be configured through URL properties stored in the #anchor part of the URL. In TOML files, they come with a very natural syntax (althrough the URL scheme is also supported). * The previously scattered EL-related state and logic is now moved to `eth1_monitor.nim` (this module will be renamed to `el_manager.nim` in a follow-up commit). State is assigned properly either to the `ELManager` or the to individual `ELConnection` objects where appropriate. The ELManager executes all Engine API requests against all attached EL nodes, in parallel. It compares their results and if there is a disagreement regarding the validity of a certain payload, this is detected and the beacon node is protected from publishing a block with a potential execution layer consensus bug in it. The BN provides metrics per EL node for the number of successful or failed requests for each type Engine API requests. If an EL node goes offline and connectivity is resoted later, we report the problem and the remedy in edge-triggered fashion. * More progress towards implementing Deneb block production in the VC and comparing the value of blocks produced by the EL and the builder API. * Adds a Makefile target for the zhejiang testnet
2023-03-05 01:40:21 +00:00
engineUrl: EngineApiUrl): T =
ELConnection(
engineUrl: engineUrl,
depositContractSyncStatus: DepositContractSyncStatus.unknown)
A hacky work-around for a web3 issue that may cause the client to go into a loop of failing requests
2020-11-17 19:50:07 +00:00
Add Eth1 deposits simulation to block_sim
2020-12-03 04:30:35 +00:00
template getOrDefault[T, E](r: Result[T, E]): T =
type TT = T
get(r, default(TT))
Migrating the deposit contract snapshot can no longer fail on start-up (#4438) The missing piece of data that had to be obtained previously from the configured EL client is now part of the network metadata baked into the binary.
2022-12-19 17:19:48 +00:00
proc init*(T: type Eth1Chain,
cfg: RuntimeConfig,
db: BeaconChainDB,
depositContractBlockNumber: uint64,
depositContractBlockHash: Eth2Digest): T =
cleanups (#3819) * avoid circular panda imports * move deposit merkleization helpers to spec/ * normalize validator signature helpers to spec names / params * remove redundant functions for remote signing
2022-06-29 16:53:59 +00:00
let
Migrating the deposit contract snapshot can no longer fail on start-up (#4438) The missing piece of data that had to be obtained previously from the configured EL client is now part of the network metadata baked into the binary.
2022-12-19 17:19:48 +00:00
(finalizedBlockHash, depositContractState) =
allow driving EL with LC (#3865) Adds the `--web3-url` launch argument to `nimbus_light_client` to enable driving the EL with the optimistic head obtained from LC sync protocol. This will keep issuing `newPayload` / `forkChoiceUpdated` requests for new blocks, marking them as optimistic. `ZERO_HASH` is reported as the finalized block for now.
2022-07-14 04:07:40 +00:00
if db != nil:
Migrating the deposit contract snapshot can no longer fail on start-up (#4438) The missing piece of data that had to be obtained previously from the configured EL client is now part of the network metadata baked into the binary.
2022-12-19 17:19:48 +00:00
let treeSnapshot = db.getDepositTreeSnapshot()
if treeSnapshot.isSome:
(treeSnapshot.get.eth1Block, treeSnapshot.get.depositContractState)
else:
let oldSnapshot = db.getUpgradableDepositSnapshot()
if oldSnapshot.isSome:
(oldSnapshot.get.eth1Block, oldSnapshot.get.depositContractState)
else:
db.putDepositTreeSnapshot DepositTreeSnapshot(
eth1Block: depositContractBlockHash,
blockHeight: depositContractBlockNumber)
(depositContractBlockHash, default(DepositContractState))
allow driving EL with LC (#3865) Adds the `--web3-url` launch argument to `nimbus_light_client` to enable driving the EL with the optimistic head obtained from LC sync protocol. This will keep issuing `newPayload` / `forkChoiceUpdated` requests for new blocks, marking them as optimistic. `ZERO_HASH` is reported as the finalized block for now.
2022-07-14 04:07:40 +00:00
else:
Migrating the deposit contract snapshot can no longer fail on start-up (#4438) The missing piece of data that had to be obtained previously from the configured EL client is now part of the network metadata baked into the binary.
2022-12-19 17:19:48 +00:00
(depositContractBlockHash, default(DepositContractState))
m = DepositsMerkleizer.init(depositContractState)
Add Eth1 deposits simulation to block_sim
2020-12-03 04:30:35 +00:00
T(db: db,
Implement split preset/config support (#2710) * Implement split preset/config support This is the initial bulk refactor to introduce runtime config values in a number of places, somewhat replacing the existing mechanism of loading network metadata. It still needs more work, this is the initial refactor that introduces runtime configuration in some of the places that need it. The PR changes the way presets and constants work, to match the spec. In particular, a "preset" now refers to the compile-time configuration while a "cfg" or "RuntimeConfig" is the dynamic part. A single binary can support either mainnet or minimal, but not both. Support for other presets has been removed completely (can be readded, in case there's need). There's a number of outstanding tasks: * `SECONDS_PER_SLOT` still needs fixing * loading custom runtime configs needs redoing * checking constants against YAML file * yeerongpilly support `build/nimbus_beacon_node --network=yeerongpilly --discv5:no --log-level=DEBUG` * load fork epoch from config * fix fork digest sent in status * nicer error string for request failures * fix tools * one more * fixup * fixup * fixup * use "standard" network definition folder in local testnet Files are loaded from their standard locations, including genesis etc, to conform to the format used in the `eth2-networks` repo. * fix launch scripts, allow unknown config values * fix base config of rest test * cleanups * bundle mainnet config using common loader * fix spec links and names * only include supported preset in binary * drop yeerongpilly, add altair-devnet-0, support boot_enr.yaml
2021-07-12 13:01:38 +00:00
cfg: cfg,
Migrating the deposit contract snapshot can no longer fail on start-up (#4438) The missing piece of data that had to be obtained previously from the configured EL client is now part of the network metadata baked into the binary.
2022-12-19 17:19:48 +00:00
finalizedBlockHash: finalizedBlockHash,
Minor post-merge cleanups (#3945) https://github.com/status-im/nimbus-eth2/pull/3944 The use of nested `awaitWithRetries` calls would have resulted in an unexpected number of retries (3x3). We now use regular `await` in outer layer to avoid the problem. https://github.com/status-im/nimbus-eth2/pull/3943 The new code has an invariant that the `headMerkleizer` field in the `Eth1Chain` is always kept in sync with the blocks stored in the chain. This invariant is now enforced better by doing the necessary merkleizer updates in the `Eth1Chain.addBlock` function, in the `Eth1Chain.init` function and in the `Eth1Chain.reset` function.
2022-08-10 12:31:10 +00:00
finalizedDepositsMerkleizer: m,
ssz: bump (#5463) more `zeroMem` avoidance, value semantics for merkleizer
2023-09-28 11:59:33 +00:00
headMerkleizer: m)
Add Eth1 deposits simulation to block_sim
2020-12-03 04:30:35 +00:00
Support for driving multiple EL nodes from a single Nimbus BN (#4465) * Support for driving multiple EL nodes from a single Nimbus BN Full list of changes: * Eth1Monitor has been renamed to ELManager to match its current responsibilities better. * The ELManager is no longer optional in the code (it won't have a nil value under any circumstances). * The support for subscribing for headers was removed as it only worked with WebSockets and contributed significant complexity while bringing only a very minor advantage. * The `--web3-url` parameter has been deprecated in favor of a new `--el` parameter. The new parameter has a reasonable default value and supports specifying a different JWT for each connection. Each connection can also be configured with a different set of responsibilities (e.g. download deposits, validate blocks and/or produce blocks). On the command-line, these properties can be configured through URL properties stored in the #anchor part of the URL. In TOML files, they come with a very natural syntax (althrough the URL scheme is also supported). * The previously scattered EL-related state and logic is now moved to `eth1_monitor.nim` (this module will be renamed to `el_manager.nim` in a follow-up commit). State is assigned properly either to the `ELManager` or the to individual `ELConnection` objects where appropriate. The ELManager executes all Engine API requests against all attached EL nodes, in parallel. It compares their results and if there is a disagreement regarding the validity of a certain payload, this is detected and the beacon node is protected from publishing a block with a potential execution layer consensus bug in it. The BN provides metrics per EL node for the number of successful or failed requests for each type Engine API requests. If an EL node goes offline and connectivity is resoted later, we report the problem and the remedy in edge-triggered fashion. * More progress towards implementing Deneb block production in the VC and comparing the value of blocks produced by the EL and the builder API. * Adds a Makefile target for the zhejiang testnet
2023-03-05 01:40:21 +00:00
proc new*(T: type ELManager,
cfg: RuntimeConfig,
depositContractBlockNumber: uint64,
depositContractBlockHash: Eth2Digest,
db: BeaconChainDB,
engineApiUrls: seq[EngineApiUrl],
eth1Network: Option[Eth1Network]): T =
let
eth1Chain = Eth1Chain.init(
cfg, db, depositContractBlockNumber, depositContractBlockHash)
Bellatrix TTD detection (#3745) * Bellatrix TTD detection * Update beacon_chain/eth1/eth1_monitor.nim Co-authored-by: Etan Kissling <etan@status.im> * Update beacon_chain/nimbus_beacon_node.nim Co-authored-by: tersec <tersec@users.noreply.github.com> Co-authored-by: Etan Kissling <etan@status.im> Co-authored-by: tersec <tersec@users.noreply.github.com>
2022-06-15 02:38:27 +00:00
Support for driving multiple EL nodes from a single Nimbus BN (#4465) * Support for driving multiple EL nodes from a single Nimbus BN Full list of changes: * Eth1Monitor has been renamed to ELManager to match its current responsibilities better. * The ELManager is no longer optional in the code (it won't have a nil value under any circumstances). * The support for subscribing for headers was removed as it only worked with WebSockets and contributed significant complexity while bringing only a very minor advantage. * The `--web3-url` parameter has been deprecated in favor of a new `--el` parameter. The new parameter has a reasonable default value and supports specifying a different JWT for each connection. Each connection can also be configured with a different set of responsibilities (e.g. download deposits, validate blocks and/or produce blocks). On the command-line, these properties can be configured through URL properties stored in the #anchor part of the URL. In TOML files, they come with a very natural syntax (althrough the URL scheme is also supported). * The previously scattered EL-related state and logic is now moved to `eth1_monitor.nim` (this module will be renamed to `el_manager.nim` in a follow-up commit). State is assigned properly either to the `ELManager` or the to individual `ELConnection` objects where appropriate. The ELManager executes all Engine API requests against all attached EL nodes, in parallel. It compares their results and if there is a disagreement regarding the validity of a certain payload, this is detected and the beacon node is protected from publishing a block with a potential execution layer consensus bug in it. The BN provides metrics per EL node for the number of successful or failed requests for each type Engine API requests. If an EL node goes offline and connectivity is resoted later, we report the problem and the remedy in edge-triggered fashion. * More progress towards implementing Deneb block production in the VC and comparing the value of blocks produced by the EL and the builder API. * Adds a Makefile target for the zhejiang testnet
2023-03-05 01:40:21 +00:00
debug "Initializing ELManager",
Local sim impovements (#4551) * Local sim impovements * Added support for running Capella and EIP-4844 simulations by downloading the correct version of Geth. * Added support for using Nimbus remote signer and Web3Signer. Use 2 out of 3 threshold signing configuration in the mainnet configuration and regular remote signing in the minimal one. * The local testnet simulation can now use a payload builder. This is currently not activated in CI due to lack of automated procedures for installing third-party relays or builders. You are adviced to use mergemock for now, but for most realistic results, we can create a simple builder based on the nimbus-eth1 codebase that will be able to propose transactions from the regular network mempool. * Start the simulation from a merged state. This would allow us to start removing pre-merge functionality such as the gossip subsciption logic. The commit also removes the merge-forcing hack installed after the TTD removal. * Consolidate all the tools used in the local simulation into a single `ncli_testnet` binary.
2023-02-23 02:10:07 +00:00
depositContractBlockNumber,
depositContractBlockHash
Support for driving multiple EL nodes from a single Nimbus BN (#4465) * Support for driving multiple EL nodes from a single Nimbus BN Full list of changes: * Eth1Monitor has been renamed to ELManager to match its current responsibilities better. * The ELManager is no longer optional in the code (it won't have a nil value under any circumstances). * The support for subscribing for headers was removed as it only worked with WebSockets and contributed significant complexity while bringing only a very minor advantage. * The `--web3-url` parameter has been deprecated in favor of a new `--el` parameter. The new parameter has a reasonable default value and supports specifying a different JWT for each connection. Each connection can also be configured with a different set of responsibilities (e.g. download deposits, validate blocks and/or produce blocks). On the command-line, these properties can be configured through URL properties stored in the #anchor part of the URL. In TOML files, they come with a very natural syntax (althrough the URL scheme is also supported). * The previously scattered EL-related state and logic is now moved to `eth1_monitor.nim` (this module will be renamed to `el_manager.nim` in a follow-up commit). State is assigned properly either to the `ELManager` or the to individual `ELConnection` objects where appropriate. The ELManager executes all Engine API requests against all attached EL nodes, in parallel. It compares their results and if there is a disagreement regarding the validity of a certain payload, this is detected and the beacon node is protected from publishing a block with a potential execution layer consensus bug in it. The BN provides metrics per EL node for the number of successful or failed requests for each type Engine API requests. If an EL node goes offline and connectivity is resoted later, we report the problem and the remedy in edge-triggered fashion. * More progress towards implementing Deneb block production in the VC and comparing the value of blocks produced by the EL and the builder API. * Adds a Makefile target for the zhejiang testnet
2023-03-05 01:40:21 +00:00
T(eth1Chain: eth1Chain,
Implement split preset/config support (#2710) * Implement split preset/config support This is the initial bulk refactor to introduce runtime config values in a number of places, somewhat replacing the existing mechanism of loading network metadata. It still needs more work, this is the initial refactor that introduces runtime configuration in some of the places that need it. The PR changes the way presets and constants work, to match the spec. In particular, a "preset" now refers to the compile-time configuration while a "cfg" or "RuntimeConfig" is the dynamic part. A single binary can support either mainnet or minimal, but not both. Support for other presets has been removed completely (can be readded, in case there's need). There's a number of outstanding tasks: * `SECONDS_PER_SLOT` still needs fixing * loading custom runtime configs needs redoing * checking constants against YAML file * yeerongpilly support `build/nimbus_beacon_node --network=yeerongpilly --discv5:no --log-level=DEBUG` * load fork epoch from config * fix fork digest sent in status * nicer error string for request failures * fix tools * one more * fixup * fixup * fixup * use "standard" network definition folder in local testnet Files are loaded from their standard locations, including genesis etc, to conform to the format used in the `eth2-networks` repo. * fix launch scripts, allow unknown config values * fix base config of rest test * cleanups * bundle mainnet config using common loader * fix spec links and names * only include supported preset in binary * drop yeerongpilly, add altair-devnet-0, support boot_enr.yaml
2021-07-12 13:01:38 +00:00
depositContractAddress: cfg.DEPOSIT_CONTRACT_ADDRESS,
Support for driving multiple EL nodes from a single Nimbus BN (#4465) * Support for driving multiple EL nodes from a single Nimbus BN Full list of changes: * Eth1Monitor has been renamed to ELManager to match its current responsibilities better. * The ELManager is no longer optional in the code (it won't have a nil value under any circumstances). * The support for subscribing for headers was removed as it only worked with WebSockets and contributed significant complexity while bringing only a very minor advantage. * The `--web3-url` parameter has been deprecated in favor of a new `--el` parameter. The new parameter has a reasonable default value and supports specifying a different JWT for each connection. Each connection can also be configured with a different set of responsibilities (e.g. download deposits, validate blocks and/or produce blocks). On the command-line, these properties can be configured through URL properties stored in the #anchor part of the URL. In TOML files, they come with a very natural syntax (althrough the URL scheme is also supported). * The previously scattered EL-related state and logic is now moved to `eth1_monitor.nim` (this module will be renamed to `el_manager.nim` in a follow-up commit). State is assigned properly either to the `ELManager` or the to individual `ELConnection` objects where appropriate. The ELManager executes all Engine API requests against all attached EL nodes, in parallel. It compares their results and if there is a disagreement regarding the validity of a certain payload, this is detected and the beacon node is protected from publishing a block with a potential execution layer consensus bug in it. The BN provides metrics per EL node for the number of successful or failed requests for each type Engine API requests. If an EL node goes offline and connectivity is resoted later, we report the problem and the remedy in edge-triggered fashion. * More progress towards implementing Deneb block production in the VC and comparing the value of blocks produced by the EL and the builder API. * Adds a Makefile target for the zhejiang testnet
2023-03-05 01:40:21 +00:00
depositContractBlockNumber: depositContractBlockNumber,
depositContractBlockHash: depositContractBlockHash.asBlockHash,
elConnections: mapIt(engineApiUrls, ELConnection.new(it)),
More complete reset of the web3 provider on each failure; Fix #2184
2020-12-15 21:59:29 +00:00
eth1Network: eth1Network,
remove TTD monitoring (#4486)
2023-01-18 14:01:49 +00:00
blocksPerLogsRequest: targetBlocksPerLogsRequest)
Eth1 validators (#393)
2019-09-09 15:59:02 +00:00
Ctrl+C now stops the BN when waiting for genesis (#1761) * Ctrl+C now stops the BN when waiting for genesis * proper shutdown when waiting for genesis if Ctrl+C is used
2020-09-28 15:19:57 +00:00
proc safeCancel(fut: var Future[void]) =
if not fut.isNil and not fut.finished:
BN+LC+SN: Fix cancellation deprecate warnings. (#5455) * Fix deprecation cancel() warnings in SN, BN, LC. * Fix missing import.
2023-09-22 11:06:27 +00:00
fut.cancelSoon()
Further reduce the number of RPC requests; Significantly faster ETH1 syncing
2020-11-03 01:21:07 +00:00
fut = nil
Ctrl+C now stops the BN when waiting for genesis (#1761) * Ctrl+C now stops the BN when waiting for genesis * proper shutdown when waiting for genesis if Ctrl+C is used
2020-09-28 15:19:57 +00:00
implement terminal-total-difficulty-override; keep kintsugi TTDs consistent (#3118) * implement terminal-total-difficulty-override; keep TTD consistent for m2 scripts/docs * use Option[uint64] instead of uint64
2021-11-25 10:53:31 +00:00
func clear(chain: var Eth1Chain) =
Fix the corrupted database state on Pyrmont nodes; Add mainnet genesis (#2056) * Handle some web3 timeouts better * Add support for developer .env files * Eth1 improvements; Mainnet genesis state Notable changes: * The deposits table have been removed from the database. The client will no longer process all deposits on start-up. * The network metadata now includes a "state snapshot" of the deposit contract. This allows the client to skip syncing deposits made prior to the snapshot (i.e. genesis). Suitable metadata added for Pyrmont and Mainnet. * The Eth1 monitor won't be started unless there are validators attached to the node. * The genesis detection code is now optional and disabled by default * Bugfix: The client should not produce blocks that will fail validation when it hasn't downloaded the latest deposits yet * Bugfix: Work around the database corruption affecting Pyrmont nodes * Remove metadata for Toledo and Medalla
2020-11-24 21:21:47 +00:00
chain.blocks.clear()
chain.blocksByHash.clear()
ssz: bump (#5463) more `zeroMem` avoidance, value semantics for merkleizer
2023-09-28 11:59:33 +00:00
chain.headMerkleizer = chain.finalizedDepositsMerkleizer
Fix #2261 Also bumps Confutils to allow setting the hidden --web3-mode param (to allow testing the eth1 syncing without validators)
2021-01-29 21:21:44 +00:00
chain.hasConsensusViolation = false
Fix the corrupted database state on Pyrmont nodes; Add mainnet genesis (#2056) * Handle some web3 timeouts better * Add support for developer .env files * Eth1 improvements; Mainnet genesis state Notable changes: * The deposits table have been removed from the database. The client will no longer process all deposits on start-up. * The network metadata now includes a "state snapshot" of the deposit contract. This allows the client to skip syncing deposits made prior to the snapshot (i.e. genesis). Suitable metadata added for Pyrmont and Mainnet. * The Eth1 monitor won't be started unless there are validators attached to the node. * The genesis detection code is now optional and disabled by default * Bugfix: The client should not produce blocks that will fail validation when it hasn't downloaded the latest deposits yet * Bugfix: Work around the database corruption affecting Pyrmont nodes * Remove metadata for Toledo and Medalla
2020-11-24 21:21:47 +00:00
Support for driving multiple EL nodes from a single Nimbus BN (#4465) * Support for driving multiple EL nodes from a single Nimbus BN Full list of changes: * Eth1Monitor has been renamed to ELManager to match its current responsibilities better. * The ELManager is no longer optional in the code (it won't have a nil value under any circumstances). * The support for subscribing for headers was removed as it only worked with WebSockets and contributed significant complexity while bringing only a very minor advantage. * The `--web3-url` parameter has been deprecated in favor of a new `--el` parameter. The new parameter has a reasonable default value and supports specifying a different JWT for each connection. Each connection can also be configured with a different set of responsibilities (e.g. download deposits, validate blocks and/or produce blocks). On the command-line, these properties can be configured through URL properties stored in the #anchor part of the URL. In TOML files, they come with a very natural syntax (althrough the URL scheme is also supported). * The previously scattered EL-related state and logic is now moved to `eth1_monitor.nim` (this module will be renamed to `el_manager.nim` in a follow-up commit). State is assigned properly either to the `ELManager` or the to individual `ELConnection` objects where appropriate. The ELManager executes all Engine API requests against all attached EL nodes, in parallel. It compares their results and if there is a disagreement regarding the validity of a certain payload, this is detected and the beacon node is protected from publishing a block with a potential execution layer consensus bug in it. The BN provides metrics per EL node for the number of successful or failed requests for each type Engine API requests. If an EL node goes offline and connectivity is resoted later, we report the problem and the remedy in edge-triggered fashion. * More progress towards implementing Deneb block production in the VC and comparing the value of blocks produced by the EL and the builder API. * Adds a Makefile target for the zhejiang testnet
2023-03-05 01:40:21 +00:00
proc doStop(m: ELManager) {.async.} =
safeCancel m.chainSyncingLoopFut
safeCancel m.exchangeTransitionConfigurationLoopFut
remove some usage of calls not required by engine API (#3761)
2022-06-17 06:32:52 +00:00
Support for driving multiple EL nodes from a single Nimbus BN (#4465) * Support for driving multiple EL nodes from a single Nimbus BN Full list of changes: * Eth1Monitor has been renamed to ELManager to match its current responsibilities better. * The ELManager is no longer optional in the code (it won't have a nil value under any circumstances). * The support for subscribing for headers was removed as it only worked with WebSockets and contributed significant complexity while bringing only a very minor advantage. * The `--web3-url` parameter has been deprecated in favor of a new `--el` parameter. The new parameter has a reasonable default value and supports specifying a different JWT for each connection. Each connection can also be configured with a different set of responsibilities (e.g. download deposits, validate blocks and/or produce blocks). On the command-line, these properties can be configured through URL properties stored in the #anchor part of the URL. In TOML files, they come with a very natural syntax (althrough the URL scheme is also supported). * The previously scattered EL-related state and logic is now moved to `eth1_monitor.nim` (this module will be renamed to `el_manager.nim` in a follow-up commit). State is assigned properly either to the `ELManager` or the to individual `ELConnection` objects where appropriate. The ELManager executes all Engine API requests against all attached EL nodes, in parallel. It compares their results and if there is a disagreement regarding the validity of a certain payload, this is detected and the beacon node is protected from publishing a block with a potential execution layer consensus bug in it. The BN provides metrics per EL node for the number of successful or failed requests for each type Engine API requests. If an EL node goes offline and connectivity is resoted later, we report the problem and the remedy in edge-triggered fashion. * More progress towards implementing Deneb block production in the VC and comparing the value of blocks produced by the EL and the builder API. * Adds a Makefile target for the zhejiang testnet
2023-03-05 01:40:21 +00:00
if m.elConnections.len > 0:
let closeConnectionFutures = mapIt(m.elConnections, close(it))
await allFutures(closeConnectionFutures)
Eth1 monitor fixes * Fix a resource leak introduced in https://github.com/status-im/nimbus-eth2/pull/3279 * Don't restart the Eth1 syncing proggress from scratch in case of monitor failures during Eth2 syncing. * Switch to the primary operator as soon as it is back online. * Log the web3 credentials in fewer places Other changes: The 'web3 test' command has been enhanced to obtain and print more data regarding the selected provider.
2022-01-31 17:28:26 +00:00
Support for driving multiple EL nodes from a single Nimbus BN (#4465) * Support for driving multiple EL nodes from a single Nimbus BN Full list of changes: * Eth1Monitor has been renamed to ELManager to match its current responsibilities better. * The ELManager is no longer optional in the code (it won't have a nil value under any circumstances). * The support for subscribing for headers was removed as it only worked with WebSockets and contributed significant complexity while bringing only a very minor advantage. * The `--web3-url` parameter has been deprecated in favor of a new `--el` parameter. The new parameter has a reasonable default value and supports specifying a different JWT for each connection. Each connection can also be configured with a different set of responsibilities (e.g. download deposits, validate blocks and/or produce blocks). On the command-line, these properties can be configured through URL properties stored in the #anchor part of the URL. In TOML files, they come with a very natural syntax (althrough the URL scheme is also supported). * The previously scattered EL-related state and logic is now moved to `eth1_monitor.nim` (this module will be renamed to `el_manager.nim` in a follow-up commit). State is assigned properly either to the `ELManager` or the to individual `ELConnection` objects where appropriate. The ELManager executes all Engine API requests against all attached EL nodes, in parallel. It compares their results and if there is a disagreement regarding the validity of a certain payload, this is detected and the beacon node is protected from publishing a block with a potential execution layer consensus bug in it. The BN provides metrics per EL node for the number of successful or failed requests for each type Engine API requests. If an EL node goes offline and connectivity is resoted later, we report the problem and the remedy in edge-triggered fashion. * More progress towards implementing Deneb block production in the VC and comparing the value of blocks produced by the EL and the builder API. * Adds a Makefile target for the zhejiang testnet
2023-03-05 01:40:21 +00:00
proc stop(m: ELManager) {.async.} =
if not m.stopFut.isNil:
More complete reset of the web3 provider on each failure; Fix #2184
2020-12-15 21:59:29 +00:00
await m.stopFut
Support for driving multiple EL nodes from a single Nimbus BN (#4465) * Support for driving multiple EL nodes from a single Nimbus BN Full list of changes: * Eth1Monitor has been renamed to ELManager to match its current responsibilities better. * The ELManager is no longer optional in the code (it won't have a nil value under any circumstances). * The support for subscribing for headers was removed as it only worked with WebSockets and contributed significant complexity while bringing only a very minor advantage. * The `--web3-url` parameter has been deprecated in favor of a new `--el` parameter. The new parameter has a reasonable default value and supports specifying a different JWT for each connection. Each connection can also be configured with a different set of responsibilities (e.g. download deposits, validate blocks and/or produce blocks). On the command-line, these properties can be configured through URL properties stored in the #anchor part of the URL. In TOML files, they come with a very natural syntax (althrough the URL scheme is also supported). * The previously scattered EL-related state and logic is now moved to `eth1_monitor.nim` (this module will be renamed to `el_manager.nim` in a follow-up commit). State is assigned properly either to the `ELManager` or the to individual `ELConnection` objects where appropriate. The ELManager executes all Engine API requests against all attached EL nodes, in parallel. It compares their results and if there is a disagreement regarding the validity of a certain payload, this is detected and the beacon node is protected from publishing a block with a potential execution layer consensus bug in it. The BN provides metrics per EL node for the number of successful or failed requests for each type Engine API requests. If an EL node goes offline and connectivity is resoted later, we report the problem and the remedy in edge-triggered fashion. * More progress towards implementing Deneb block production in the VC and comparing the value of blocks produced by the EL and the builder API. * Adds a Makefile target for the zhejiang testnet
2023-03-05 01:40:21 +00:00
else:
m.stopFut = m.doStop()
More complete reset of the web3 provider on each failure; Fix #2184
2020-12-15 21:59:29 +00:00
await m.stopFut
Support for driving multiple EL nodes from a single Nimbus BN (#4465) * Support for driving multiple EL nodes from a single Nimbus BN Full list of changes: * Eth1Monitor has been renamed to ELManager to match its current responsibilities better. * The ELManager is no longer optional in the code (it won't have a nil value under any circumstances). * The support for subscribing for headers was removed as it only worked with WebSockets and contributed significant complexity while bringing only a very minor advantage. * The `--web3-url` parameter has been deprecated in favor of a new `--el` parameter. The new parameter has a reasonable default value and supports specifying a different JWT for each connection. Each connection can also be configured with a different set of responsibilities (e.g. download deposits, validate blocks and/or produce blocks). On the command-line, these properties can be configured through URL properties stored in the #anchor part of the URL. In TOML files, they come with a very natural syntax (althrough the URL scheme is also supported). * The previously scattered EL-related state and logic is now moved to `eth1_monitor.nim` (this module will be renamed to `el_manager.nim` in a follow-up commit). State is assigned properly either to the `ELManager` or the to individual `ELConnection` objects where appropriate. The ELManager executes all Engine API requests against all attached EL nodes, in parallel. It compares their results and if there is a disagreement regarding the validity of a certain payload, this is detected and the beacon node is protected from publishing a block with a potential execution layer consensus bug in it. The BN provides metrics per EL node for the number of successful or failed requests for each type Engine API requests. If an EL node goes offline and connectivity is resoted later, we report the problem and the remedy in edge-triggered fashion. * More progress towards implementing Deneb block production in the VC and comparing the value of blocks produced by the EL and the builder API. * Adds a Makefile target for the zhejiang testnet
2023-03-05 01:40:21 +00:00
m.stopFut = nil
More complete reset of the web3 provider on each failure; Fix #2184
2020-12-15 21:59:29 +00:00
Tentative fix for the Pyrmont Eth1 voting issue
2020-11-19 17:19:03 +00:00
const
Fix the corrupted database state on Pyrmont nodes; Add mainnet genesis (#2056) * Handle some web3 timeouts better * Add support for developer .env files * Eth1 improvements; Mainnet genesis state Notable changes: * The deposits table have been removed from the database. The client will no longer process all deposits on start-up. * The network metadata now includes a "state snapshot" of the deposit contract. This allows the client to skip syncing deposits made prior to the snapshot (i.e. genesis). Suitable metadata added for Pyrmont and Mainnet. * The Eth1 monitor won't be started unless there are validators attached to the node. * The genesis detection code is now optional and disabled by default * Bugfix: The client should not produce blocks that will fail validation when it hasn't downloaded the latest deposits yet * Bugfix: Work around the database corruption affecting Pyrmont nodes * Remove metadata for Toledo and Medalla
2020-11-24 21:21:47 +00:00
votedBlocksSafetyMargin = 50
Tentative fix for the Pyrmont Eth1 voting issue
2020-11-19 17:19:03 +00:00
Redo fix #4588
2023-03-10 17:40:27 +00:00
func earliestBlockOfInterest(
m: ELManager,
latestEth1BlockNumber: Eth1BlockNumber): Eth1BlockNumber =
let blocksOfInterestRange =
SLOTS_PER_ETH1_VOTING_PERIOD +
(2 * m.cfg.ETH1_FOLLOW_DISTANCE) +
votedBlocksSafetyMargin
if latestEth1BlockNumber > blocksOfInterestRange:
latestEth1BlockNumber - blocksOfInterestRange
else:
0
Tentative fix for the Pyrmont Eth1 voting issue
2020-11-19 17:19:03 +00:00
Support for driving multiple EL nodes from a single Nimbus BN (#4465) * Support for driving multiple EL nodes from a single Nimbus BN Full list of changes: * Eth1Monitor has been renamed to ELManager to match its current responsibilities better. * The ELManager is no longer optional in the code (it won't have a nil value under any circumstances). * The support for subscribing for headers was removed as it only worked with WebSockets and contributed significant complexity while bringing only a very minor advantage. * The `--web3-url` parameter has been deprecated in favor of a new `--el` parameter. The new parameter has a reasonable default value and supports specifying a different JWT for each connection. Each connection can also be configured with a different set of responsibilities (e.g. download deposits, validate blocks and/or produce blocks). On the command-line, these properties can be configured through URL properties stored in the #anchor part of the URL. In TOML files, they come with a very natural syntax (althrough the URL scheme is also supported). * The previously scattered EL-related state and logic is now moved to `eth1_monitor.nim` (this module will be renamed to `el_manager.nim` in a follow-up commit). State is assigned properly either to the `ELManager` or the to individual `ELConnection` objects where appropriate. The ELManager executes all Engine API requests against all attached EL nodes, in parallel. It compares their results and if there is a disagreement regarding the validity of a certain payload, this is detected and the beacon node is protected from publishing a block with a potential execution layer consensus bug in it. The BN provides metrics per EL node for the number of successful or failed requests for each type Engine API requests. If an EL node goes offline and connectivity is resoted later, we report the problem and the remedy in edge-triggered fashion. * More progress towards implementing Deneb block production in the VC and comparing the value of blocks produced by the EL and the builder API. * Adds a Makefile target for the zhejiang testnet
2023-03-05 01:40:21 +00:00
proc syncBlockRange(m: ELManager,
More metrics for the EL monitor (#4707) * `engine_api_response_time` provides a histogram for the Engine API response times for each unique pair ot URL and request type. * All engine API requests are now tracked Other changes: The client will no longer exit on start-up if it fails to connect to a properly configured EL node.
2023-03-09 17:29:04 +00:00
connection: ELConnection,
Support for driving multiple EL nodes from a single Nimbus BN (#4465) * Support for driving multiple EL nodes from a single Nimbus BN Full list of changes: * Eth1Monitor has been renamed to ELManager to match its current responsibilities better. * The ELManager is no longer optional in the code (it won't have a nil value under any circumstances). * The support for subscribing for headers was removed as it only worked with WebSockets and contributed significant complexity while bringing only a very minor advantage. * The `--web3-url` parameter has been deprecated in favor of a new `--el` parameter. The new parameter has a reasonable default value and supports specifying a different JWT for each connection. Each connection can also be configured with a different set of responsibilities (e.g. download deposits, validate blocks and/or produce blocks). On the command-line, these properties can be configured through URL properties stored in the #anchor part of the URL. In TOML files, they come with a very natural syntax (althrough the URL scheme is also supported). * The previously scattered EL-related state and logic is now moved to `eth1_monitor.nim` (this module will be renamed to `el_manager.nim` in a follow-up commit). State is assigned properly either to the `ELManager` or the to individual `ELConnection` objects where appropriate. The ELManager executes all Engine API requests against all attached EL nodes, in parallel. It compares their results and if there is a disagreement regarding the validity of a certain payload, this is detected and the beacon node is protected from publishing a block with a potential execution layer consensus bug in it. The BN provides metrics per EL node for the number of successful or failed requests for each type Engine API requests. If an EL node goes offline and connectivity is resoted later, we report the problem and the remedy in edge-triggered fashion. * More progress towards implementing Deneb block production in the VC and comparing the value of blocks produced by the EL and the builder API. * Adds a Makefile target for the zhejiang testnet
2023-03-05 01:40:21 +00:00
rpcClient: RpcClient,
depositContract: Sender[DepositContract],
Tentative fix for the Pyrmont Eth1 voting issue
2020-11-19 17:19:03 +00:00
fromBlock, toBlock,
More complete reset of the web3 provider on each failure; Fix #2184
2020-12-15 21:59:29 +00:00
fullSyncFromBlock: Eth1BlockNumber) {.gcsafe, async.} =
Support for driving multiple EL nodes from a single Nimbus BN (#4465) * Support for driving multiple EL nodes from a single Nimbus BN Full list of changes: * Eth1Monitor has been renamed to ELManager to match its current responsibilities better. * The ELManager is no longer optional in the code (it won't have a nil value under any circumstances). * The support for subscribing for headers was removed as it only worked with WebSockets and contributed significant complexity while bringing only a very minor advantage. * The `--web3-url` parameter has been deprecated in favor of a new `--el` parameter. The new parameter has a reasonable default value and supports specifying a different JWT for each connection. Each connection can also be configured with a different set of responsibilities (e.g. download deposits, validate blocks and/or produce blocks). On the command-line, these properties can be configured through URL properties stored in the #anchor part of the URL. In TOML files, they come with a very natural syntax (althrough the URL scheme is also supported). * The previously scattered EL-related state and logic is now moved to `eth1_monitor.nim` (this module will be renamed to `el_manager.nim` in a follow-up commit). State is assigned properly either to the `ELManager` or the to individual `ELConnection` objects where appropriate. The ELManager executes all Engine API requests against all attached EL nodes, in parallel. It compares their results and if there is a disagreement regarding the validity of a certain payload, this is detected and the beacon node is protected from publishing a block with a potential execution layer consensus bug in it. The BN provides metrics per EL node for the number of successful or failed requests for each type Engine API requests. If an EL node goes offline and connectivity is resoted later, we report the problem and the remedy in edge-triggered fashion. * More progress towards implementing Deneb block production in the VC and comparing the value of blocks produced by the EL and the builder API. * Adds a Makefile target for the zhejiang testnet
2023-03-05 01:40:21 +00:00
doAssert m.eth1Chain.blocks.len > 0
Tentative fix for the Pyrmont Eth1 voting issue
2020-11-19 17:19:03 +00:00
Further reduce the number of RPC requests; Significantly faster ETH1 syncing
2020-11-03 01:21:07 +00:00
var currentBlock = fromBlock
while currentBlock <= toBlock:
Bugfix: the client can miss the genesis event in the absence of new deposits
2020-11-14 20:51:50 +00:00
var
depositLogs: JsonNode = nil
maxBlockNumberRequested: Eth1BlockNumber
Add backoff for the deposit logs requests as well; More debug logs
2020-12-01 23:35:07 +00:00
backoff = 100
Spec-compliant implementation of Eth1 monitoring; Eth1-enabled local sim BEWARE! This commit will trigger a stack overflow during local sim
2020-03-24 11:13:07 +00:00
Further reduce the number of RPC requests; Significantly faster ETH1 syncing
2020-11-03 01:21:07 +00:00
while true:
track request chunk size across EL reconnects (#3960) When the EL connection is interrupted, deposits are once more requested in chunks of 5000 blocks. This is a problem when the response takes over a minute to produce and consistently times out as followup requests with lower chunk sizes may no longer work after a request was canceled, e.g., when using Geth with websockets. By keeping track of `blocksPerRequest` across EL reconnections, it is possible to recover from this by avoiding to continuously repeat the initial request with the full 5000 blocks. Also cleans up one more "retry of retry" instance; `DataProviderTimeout` is a `CatchableError` and already handled by the existing retry logic.
2022-08-12 13:51:33 +00:00
maxBlockNumberRequested =
min(toBlock, currentBlock + m.blocksPerLogsRequest - 1)
Spec-compliant implementation of Eth1 monitoring; Eth1-enabled local sim BEWARE! This commit will trigger a stack overflow during local sim
2020-03-24 11:13:07 +00:00
Fix the corrupted database state on Pyrmont nodes; Add mainnet genesis (#2056) * Handle some web3 timeouts better * Add support for developer .env files * Eth1 improvements; Mainnet genesis state Notable changes: * The deposits table have been removed from the database. The client will no longer process all deposits on start-up. * The network metadata now includes a "state snapshot" of the deposit contract. This allows the client to skip syncing deposits made prior to the snapshot (i.e. genesis). Suitable metadata added for Pyrmont and Mainnet. * The Eth1 monitor won't be started unless there are validators attached to the node. * The genesis detection code is now optional and disabled by default * Bugfix: The client should not produce blocks that will fail validation when it hasn't downloaded the latest deposits yet * Bugfix: Work around the database corruption affecting Pyrmont nodes * Remove metadata for Toledo and Medalla
2020-11-24 21:21:47 +00:00
debug "Obtaining deposit log events",
fromBlock = currentBlock,
Add backoff for the deposit logs requests as well; More debug logs
2020-12-01 23:35:07 +00:00
toBlock = maxBlockNumberRequested,
backoff
Clean 20200205 (#729) * beacon node code cleanup * rudimentary error checking on mainnet monitor * start client even when sending deposit * work around missing block number exception * connect to testnet with web3 url * pretty-print digests in json
2020-02-07 07:13:38 +00:00
Fix the corrupted database state on Pyrmont nodes; Add mainnet genesis (#2056) * Handle some web3 timeouts better * Add support for developer .env files * Eth1 improvements; Mainnet genesis state Notable changes: * The deposits table have been removed from the database. The client will no longer process all deposits on start-up. * The network metadata now includes a "state snapshot" of the deposit contract. This allows the client to skip syncing deposits made prior to the snapshot (i.e. genesis). Suitable metadata added for Pyrmont and Mainnet. * The Eth1 monitor won't be started unless there are validators attached to the node. * The genesis detection code is now optional and disabled by default * Bugfix: The client should not produce blocks that will fail validation when it hasn't downloaded the latest deposits yet * Bugfix: Work around the database corruption affecting Pyrmont nodes * Remove metadata for Toledo and Medalla
2020-11-24 21:21:47 +00:00
debug.logTime "Deposit logs obtained":
Reduce the rate of requesting deposits to lower the risk of triggering the Infura rate limits
2020-12-01 11:14:32 +00:00
# Reduce all request rate until we have a more general solution
# for dealing with Infura's rate limits
Add backoff for the deposit logs requests as well; More debug logs
2020-12-01 23:35:07 +00:00
await sleepAsync(milliseconds(backoff))
More metrics for the EL monitor (#4707) * `engine_api_response_time` provides a histogram for the Engine API response times for each unique pair ot URL and request type. * All engine API requests are now tracked Other changes: The client will no longer exit on start-up if it fails to connect to a properly configured EL node.
2023-03-09 17:29:04 +00:00
let
startTime = Moment.now
deadline = sleepAsync 30.seconds
jsonLogsFut = depositContract.getJsonLogs(
DepositEvent,
fromBlock = some blockId(currentBlock),
toBlock = some blockId(maxBlockNumberRequested))
connection.trackEngineApiRequest(
jsonLogsFut, "getLogs", startTime, deadline)
Fix the corrupted database state on Pyrmont nodes; Add mainnet genesis (#2056) * Handle some web3 timeouts better * Add support for developer .env files * Eth1 improvements; Mainnet genesis state Notable changes: * The deposits table have been removed from the database. The client will no longer process all deposits on start-up. * The network metadata now includes a "state snapshot" of the deposit contract. This allows the client to skip syncing deposits made prior to the snapshot (i.e. genesis). Suitable metadata added for Pyrmont and Mainnet. * The Eth1 monitor won't be started unless there are validators attached to the node. * The genesis detection code is now optional and disabled by default * Bugfix: The client should not produce blocks that will fail validation when it hasn't downloaded the latest deposits yet * Bugfix: Work around the database corruption affecting Pyrmont nodes * Remove metadata for Toledo and Medalla
2020-11-24 21:21:47 +00:00
depositLogs = try:
track request chunk size across EL reconnects (#3960) When the EL connection is interrupted, deposits are once more requested in chunks of 5000 blocks. This is a problem when the response takes over a minute to produce and consistently times out as followup requests with lower chunk sizes may no longer work after a request was canceled, e.g., when using Geth with websockets. By keeping track of `blocksPerRequest` across EL reconnections, it is possible to recover from this by avoiding to continuously repeat the initial request with the full 5000 blocks. Also cleans up one more "retry of retry" instance; `DataProviderTimeout` is a `CatchableError` and already handled by the existing retry logic.
2022-08-12 13:51:33 +00:00
# Downloading large amounts of deposits may take several minutes
More metrics for the EL monitor (#4707) * `engine_api_response_time` provides a histogram for the Engine API response times for each unique pair ot URL and request type. * All engine API requests are now tracked Other changes: The client will no longer exit on start-up if it fails to connect to a properly configured EL node.
2023-03-09 17:29:04 +00:00
awaitWithTimeout(jsonLogsFut, deadline):
track request chunk size across EL reconnects (#3960) When the EL connection is interrupted, deposits are once more requested in chunks of 5000 blocks. This is a problem when the response takes over a minute to produce and consistently times out as followup requests with lower chunk sizes may no longer work after a request was canceled, e.g., when using Geth with websockets. By keeping track of `blocksPerRequest` across EL reconnections, it is possible to recover from this by avoiding to continuously repeat the initial request with the full 5000 blocks. Also cleans up one more "retry of retry" instance; `DataProviderTimeout` is a `CatchableError` and already handled by the existing retry logic.
2022-08-12 13:51:33 +00:00
raise newException(DataProviderTimeout,
Fix the corrupted database state on Pyrmont nodes; Add mainnet genesis (#2056) * Handle some web3 timeouts better * Add support for developer .env files * Eth1 improvements; Mainnet genesis state Notable changes: * The deposits table have been removed from the database. The client will no longer process all deposits on start-up. * The network metadata now includes a "state snapshot" of the deposit contract. This allows the client to skip syncing deposits made prior to the snapshot (i.e. genesis). Suitable metadata added for Pyrmont and Mainnet. * The Eth1 monitor won't be started unless there are validators attached to the node. * The genesis detection code is now optional and disabled by default * Bugfix: The client should not produce blocks that will fail validation when it hasn't downloaded the latest deposits yet * Bugfix: Work around the database corruption affecting Pyrmont nodes * Remove metadata for Toledo and Medalla
2020-11-24 21:21:47 +00:00
"Request time out while obtaining json logs")
except CatchableError as err:
Add backoff for the deposit logs requests as well; More debug logs
2020-12-01 23:35:07 +00:00
debug "Request for deposit logs failed", err = err.msg
Add some useful Eth1 monitor metrics
2020-12-09 22:44:59 +00:00
inc failed_web3_requests
Add backoff for the deposit logs requests as well; More debug logs
2020-12-01 23:35:07 +00:00
backoff = (backoff * 3) div 2
track request chunk size across EL reconnects (#3960) When the EL connection is interrupted, deposits are once more requested in chunks of 5000 blocks. This is a problem when the response takes over a minute to produce and consistently times out as followup requests with lower chunk sizes may no longer work after a request was canceled, e.g., when using Geth with websockets. By keeping track of `blocksPerRequest` across EL reconnections, it is possible to recover from this by avoiding to continuously repeat the initial request with the full 5000 blocks. Also cleans up one more "retry of retry" instance; `DataProviderTimeout` is a `CatchableError` and already handled by the existing retry logic.
2022-08-12 13:51:33 +00:00
m.blocksPerLogsRequest = m.blocksPerLogsRequest div 2
if m.blocksPerLogsRequest == 0:
m.blocksPerLogsRequest = 1
raise err
continue
m.blocksPerLogsRequest = min(
(m.blocksPerLogsRequest * 3 + 1) div 2,
targetBlocksPerLogsRequest)
Further reduce the number of RPC requests; Significantly faster ETH1 syncing
2020-11-03 01:21:07 +00:00
Fix the corrupted database state on Pyrmont nodes; Add mainnet genesis (#2056) * Handle some web3 timeouts better * Add support for developer .env files * Eth1 improvements; Mainnet genesis state Notable changes: * The deposits table have been removed from the database. The client will no longer process all deposits on start-up. * The network metadata now includes a "state snapshot" of the deposit contract. This allows the client to skip syncing deposits made prior to the snapshot (i.e. genesis). Suitable metadata added for Pyrmont and Mainnet. * The Eth1 monitor won't be started unless there are validators attached to the node. * The genesis detection code is now optional and disabled by default * Bugfix: The client should not produce blocks that will fail validation when it hasn't downloaded the latest deposits yet * Bugfix: Work around the database corruption affecting Pyrmont nodes * Remove metadata for Toledo and Medalla
2020-11-24 21:21:47 +00:00
currentBlock = maxBlockNumberRequested + 1
break
Reduce the rate of requesting deposits to lower the risk of triggering the Infura rate limits
2020-12-01 11:14:32 +00:00
let blocksWithDeposits = depositEventsToBlocks(depositLogs)
Fix the corrupted database state on Pyrmont nodes; Add mainnet genesis (#2056) * Handle some web3 timeouts better * Add support for developer .env files * Eth1 improvements; Mainnet genesis state Notable changes: * The deposits table have been removed from the database. The client will no longer process all deposits on start-up. * The network metadata now includes a "state snapshot" of the deposit contract. This allows the client to skip syncing deposits made prior to the snapshot (i.e. genesis). Suitable metadata added for Pyrmont and Mainnet. * The Eth1 monitor won't be started unless there are validators attached to the node. * The genesis detection code is now optional and disabled by default * Bugfix: The client should not produce blocks that will fail validation when it hasn't downloaded the latest deposits yet * Bugfix: Work around the database corruption affecting Pyrmont nodes * Remove metadata for Toledo and Medalla
2020-11-24 21:21:47 +00:00
for i in 0 ..< blocksWithDeposits.len:
let blk = blocksWithDeposits[i]
Tentative fix for the Pyrmont Eth1 voting issue
2020-11-19 17:19:03 +00:00
if blk.number > fullSyncFromBlock:
Reduce the number of block timestamp requests The timestamps are only relevant within the candidate block range.
2023-03-14 11:46:22 +00:00
await fetchTimestamp(connection, rpcClient, blk)
Support for driving multiple EL nodes from a single Nimbus BN (#4465) * Support for driving multiple EL nodes from a single Nimbus BN Full list of changes: * Eth1Monitor has been renamed to ELManager to match its current responsibilities better. * The ELManager is no longer optional in the code (it won't have a nil value under any circumstances). * The support for subscribing for headers was removed as it only worked with WebSockets and contributed significant complexity while bringing only a very minor advantage. * The `--web3-url` parameter has been deprecated in favor of a new `--el` parameter. The new parameter has a reasonable default value and supports specifying a different JWT for each connection. Each connection can also be configured with a different set of responsibilities (e.g. download deposits, validate blocks and/or produce blocks). On the command-line, these properties can be configured through URL properties stored in the #anchor part of the URL. In TOML files, they come with a very natural syntax (althrough the URL scheme is also supported). * The previously scattered EL-related state and logic is now moved to `eth1_monitor.nim` (this module will be renamed to `el_manager.nim` in a follow-up commit). State is assigned properly either to the `ELManager` or the to individual `ELConnection` objects where appropriate. The ELManager executes all Engine API requests against all attached EL nodes, in parallel. It compares their results and if there is a disagreement regarding the validity of a certain payload, this is detected and the beacon node is protected from publishing a block with a potential execution layer consensus bug in it. The BN provides metrics per EL node for the number of successful or failed requests for each type Engine API requests. If an EL node goes offline and connectivity is resoted later, we report the problem and the remedy in edge-triggered fashion. * More progress towards implementing Deneb block production in the VC and comparing the value of blocks produced by the EL and the builder API. * Adds a Makefile target for the zhejiang testnet
2023-03-05 01:40:21 +00:00
let lastBlock = m.eth1Chain.blocks.peekLast
Tentative fix for the Pyrmont Eth1 voting issue
2020-11-19 17:19:03 +00:00
for n in max(lastBlock.number + 1, fullSyncFromBlock) ..< blk.number:
Add backoff for the deposit logs requests as well; More debug logs
2020-12-01 23:35:07 +00:00
debug "Obtaining block without deposits", blockNum = n
More metrics for the EL monitor (#4707) * `engine_api_response_time` provides a histogram for the Engine API response times for each unique pair ot URL and request type. * All engine API requests are now tracked Other changes: The client will no longer exit on start-up if it fails to connect to a properly configured EL node.
2023-03-09 17:29:04 +00:00
let blockWithoutDeposits = connection.trackedRequestWithTimeout(
"getBlockByNumber",
Support for driving multiple EL nodes from a single Nimbus BN (#4465) * Support for driving multiple EL nodes from a single Nimbus BN Full list of changes: * Eth1Monitor has been renamed to ELManager to match its current responsibilities better. * The ELManager is no longer optional in the code (it won't have a nil value under any circumstances). * The support for subscribing for headers was removed as it only worked with WebSockets and contributed significant complexity while bringing only a very minor advantage. * The `--web3-url` parameter has been deprecated in favor of a new `--el` parameter. The new parameter has a reasonable default value and supports specifying a different JWT for each connection. Each connection can also be configured with a different set of responsibilities (e.g. download deposits, validate blocks and/or produce blocks). On the command-line, these properties can be configured through URL properties stored in the #anchor part of the URL. In TOML files, they come with a very natural syntax (althrough the URL scheme is also supported). * The previously scattered EL-related state and logic is now moved to `eth1_monitor.nim` (this module will be renamed to `el_manager.nim` in a follow-up commit). State is assigned properly either to the `ELManager` or the to individual `ELConnection` objects where appropriate. The ELManager executes all Engine API requests against all attached EL nodes, in parallel. It compares their results and if there is a disagreement regarding the validity of a certain payload, this is detected and the beacon node is protected from publishing a block with a potential execution layer consensus bug in it. The BN provides metrics per EL node for the number of successful or failed requests for each type Engine API requests. If an EL node goes offline and connectivity is resoted later, we report the problem and the remedy in edge-triggered fashion. * More progress towards implementing Deneb block production in the VC and comparing the value of blocks produced by the EL and the builder API. * Adds a Makefile target for the zhejiang testnet
2023-03-05 01:40:21 +00:00
rpcClient.getBlockByNumber(n),
web3RequestsTimeout)
Retry all web3 requests up to 3 times with exponential backoff
2020-12-01 21:20:28 +00:00
Support for driving multiple EL nodes from a single Nimbus BN (#4465) * Support for driving multiple EL nodes from a single Nimbus BN Full list of changes: * Eth1Monitor has been renamed to ELManager to match its current responsibilities better. * The ELManager is no longer optional in the code (it won't have a nil value under any circumstances). * The support for subscribing for headers was removed as it only worked with WebSockets and contributed significant complexity while bringing only a very minor advantage. * The `--web3-url` parameter has been deprecated in favor of a new `--el` parameter. The new parameter has a reasonable default value and supports specifying a different JWT for each connection. Each connection can also be configured with a different set of responsibilities (e.g. download deposits, validate blocks and/or produce blocks). On the command-line, these properties can be configured through URL properties stored in the #anchor part of the URL. In TOML files, they come with a very natural syntax (althrough the URL scheme is also supported). * The previously scattered EL-related state and logic is now moved to `eth1_monitor.nim` (this module will be renamed to `el_manager.nim` in a follow-up commit). State is assigned properly either to the `ELManager` or the to individual `ELConnection` objects where appropriate. The ELManager executes all Engine API requests against all attached EL nodes, in parallel. It compares their results and if there is a disagreement regarding the validity of a certain payload, this is detected and the beacon node is protected from publishing a block with a potential execution layer consensus bug in it. The BN provides metrics per EL node for the number of successful or failed requests for each type Engine API requests. If an EL node goes offline and connectivity is resoted later, we report the problem and the remedy in edge-triggered fashion. * More progress towards implementing Deneb block production in the VC and comparing the value of blocks produced by the EL and the builder API. * Adds a Makefile target for the zhejiang testnet
2023-03-05 01:40:21 +00:00
m.eth1Chain.addBlock(
Tentative fix for the Pyrmont Eth1 voting issue
2020-11-19 17:19:03 +00:00
lastBlock.makeSuccessorWithoutDeposits(blockWithoutDeposits))
Add some useful Eth1 monitor metrics
2020-12-09 22:44:59 +00:00
eth1_synced_head.set blockWithoutDeposits.number.toGaugeValue
Tentative fix for the Pyrmont Eth1 voting issue
2020-11-19 17:19:03 +00:00
Support for driving multiple EL nodes from a single Nimbus BN (#4465) * Support for driving multiple EL nodes from a single Nimbus BN Full list of changes: * Eth1Monitor has been renamed to ELManager to match its current responsibilities better. * The ELManager is no longer optional in the code (it won't have a nil value under any circumstances). * The support for subscribing for headers was removed as it only worked with WebSockets and contributed significant complexity while bringing only a very minor advantage. * The `--web3-url` parameter has been deprecated in favor of a new `--el` parameter. The new parameter has a reasonable default value and supports specifying a different JWT for each connection. Each connection can also be configured with a different set of responsibilities (e.g. download deposits, validate blocks and/or produce blocks). On the command-line, these properties can be configured through URL properties stored in the #anchor part of the URL. In TOML files, they come with a very natural syntax (althrough the URL scheme is also supported). * The previously scattered EL-related state and logic is now moved to `eth1_monitor.nim` (this module will be renamed to `el_manager.nim` in a follow-up commit). State is assigned properly either to the `ELManager` or the to individual `ELConnection` objects where appropriate. The ELManager executes all Engine API requests against all attached EL nodes, in parallel. It compares their results and if there is a disagreement regarding the validity of a certain payload, this is detected and the beacon node is protected from publishing a block with a potential execution layer consensus bug in it. The BN provides metrics per EL node for the number of successful or failed requests for each type Engine API requests. If an EL node goes offline and connectivity is resoted later, we report the problem and the remedy in edge-triggered fashion. * More progress towards implementing Deneb block production in the VC and comparing the value of blocks produced by the EL and the builder API. * Adds a Makefile target for the zhejiang testnet
2023-03-05 01:40:21 +00:00
m.eth1Chain.addBlock blk
Add some useful Eth1 monitor metrics
2020-12-09 22:44:59 +00:00
eth1_synced_head.set blk.number.toGaugeValue
Fix the corrupted database state on Pyrmont nodes; Add mainnet genesis (#2056) * Handle some web3 timeouts better * Add support for developer .env files * Eth1 improvements; Mainnet genesis state Notable changes: * The deposits table have been removed from the database. The client will no longer process all deposits on start-up. * The network metadata now includes a "state snapshot" of the deposit contract. This allows the client to skip syncing deposits made prior to the snapshot (i.e. genesis). Suitable metadata added for Pyrmont and Mainnet. * The Eth1 monitor won't be started unless there are validators attached to the node. * The genesis detection code is now optional and disabled by default * Bugfix: The client should not produce blocks that will fail validation when it hasn't downloaded the latest deposits yet * Bugfix: Work around the database corruption affecting Pyrmont nodes * Remove metadata for Toledo and Medalla
2020-11-24 21:21:47 +00:00
if blocksWithDeposits.len > 0:
let lastIdx = blocksWithDeposits.len - 1
template lastBlock: auto = blocksWithDeposits[lastIdx]
Further reduce the number of RPC requests; Significantly faster ETH1 syncing
2020-11-03 01:21:07 +00:00
Fix the corrupted database state on Pyrmont nodes; Add mainnet genesis (#2056) * Handle some web3 timeouts better * Add support for developer .env files * Eth1 improvements; Mainnet genesis state Notable changes: * The deposits table have been removed from the database. The client will no longer process all deposits on start-up. * The network metadata now includes a "state snapshot" of the deposit contract. This allows the client to skip syncing deposits made prior to the snapshot (i.e. genesis). Suitable metadata added for Pyrmont and Mainnet. * The Eth1 monitor won't be started unless there are validators attached to the node. * The genesis detection code is now optional and disabled by default * Bugfix: The client should not produce blocks that will fail validation when it hasn't downloaded the latest deposits yet * Bugfix: Work around the database corruption affecting Pyrmont nodes * Remove metadata for Toledo and Medalla
2020-11-24 21:21:47 +00:00
let status = when hasDepositRootChecks:
More metrics for the EL monitor (#4707) * `engine_api_response_time` provides a histogram for the Engine API response times for each unique pair ot URL and request type. * All engine API requests are now tracked Other changes: The client will no longer exit on start-up if it fails to connect to a properly configured EL node.
2023-03-09 17:29:04 +00:00
rpcClient.fetchDepositContractData(depositContract, lastBlock)
Fix the corrupted database state on Pyrmont nodes; Add mainnet genesis (#2056) * Handle some web3 timeouts better * Add support for developer .env files * Eth1 improvements; Mainnet genesis state Notable changes: * The deposits table have been removed from the database. The client will no longer process all deposits on start-up. * The network metadata now includes a "state snapshot" of the deposit contract. This allows the client to skip syncing deposits made prior to the snapshot (i.e. genesis). Suitable metadata added for Pyrmont and Mainnet. * The Eth1 monitor won't be started unless there are validators attached to the node. * The genesis detection code is now optional and disabled by default * Bugfix: The client should not produce blocks that will fail validation when it hasn't downloaded the latest deposits yet * Bugfix: Work around the database corruption affecting Pyrmont nodes * Remove metadata for Toledo and Medalla
2020-11-24 21:21:47 +00:00
else:
DepositRootUnavailable
Further reduce the number of RPC requests; Significantly faster ETH1 syncing
2020-11-03 01:21:07 +00:00
when hasDepositRootChecks:
Fix the corrupted database state on Pyrmont nodes; Add mainnet genesis (#2056) * Handle some web3 timeouts better * Add support for developer .env files * Eth1 improvements; Mainnet genesis state Notable changes: * The deposits table have been removed from the database. The client will no longer process all deposits on start-up. * The network metadata now includes a "state snapshot" of the deposit contract. This allows the client to skip syncing deposits made prior to the snapshot (i.e. genesis). Suitable metadata added for Pyrmont and Mainnet. * The Eth1 monitor won't be started unless there are validators attached to the node. * The genesis detection code is now optional and disabled by default * Bugfix: The client should not produce blocks that will fail validation when it hasn't downloaded the latest deposits yet * Bugfix: Work around the database corruption affecting Pyrmont nodes * Remove metadata for Toledo and Medalla
2020-11-24 21:21:47 +00:00
debug "Deposit contract state verified",
status = $status,
Minor post-merge cleanups (#3945) https://github.com/status-im/nimbus-eth2/pull/3944 The use of nested `awaitWithRetries` calls would have resulted in an unexpected number of retries (3x3). We now use regular `await` in outer layer to avoid the problem. https://github.com/status-im/nimbus-eth2/pull/3943 The new code has an invariant that the `headMerkleizer` field in the `Eth1Chain` is always kept in sync with the blocks stored in the chain. This invariant is now enforced better by doing the necessary merkleizer updates in the `Eth1Chain.addBlock` function, in the `Eth1Chain.init` function and in the `Eth1Chain.reset` function.
2022-08-10 12:31:10 +00:00
ourCount = lastBlock.depositCount,
ourRoot = lastBlock.depositRoot
Further reduce the number of RPC requests; Significantly faster ETH1 syncing
2020-11-03 01:21:07 +00:00
Fix the corrupted database state on Pyrmont nodes; Add mainnet genesis (#2056) * Handle some web3 timeouts better * Add support for developer .env files * Eth1 improvements; Mainnet genesis state Notable changes: * The deposits table have been removed from the database. The client will no longer process all deposits on start-up. * The network metadata now includes a "state snapshot" of the deposit contract. This allows the client to skip syncing deposits made prior to the snapshot (i.e. genesis). Suitable metadata added for Pyrmont and Mainnet. * The Eth1 monitor won't be started unless there are validators attached to the node. * The genesis detection code is now optional and disabled by default * Bugfix: The client should not produce blocks that will fail validation when it hasn't downloaded the latest deposits yet * Bugfix: Work around the database corruption affecting Pyrmont nodes * Remove metadata for Toledo and Medalla
2020-11-24 21:21:47 +00:00
case status
of DepositRootIncorrect, DepositCountIncorrect:
raise newException(CorruptDataProvider,
"The deposit log events disagree with the deposit contract state")
else:
discard
Further reduce the number of RPC requests; Significantly faster ETH1 syncing
2020-11-03 01:21:07 +00:00
Assorted logging improvements (#3237) * log doppelganger detection when it activates and when it causes missed duties * less prominent eth1 sync progress * log in-progress sync at notice only when actually missing duties * better detail in replay log * don't log finalization checkpoints - this is quite verbose when syncing and already included in "Slot start"
2022-01-03 21:18:49 +00:00
info "Eth1 sync progress",
Fix the corrupted database state on Pyrmont nodes; Add mainnet genesis (#2056) * Handle some web3 timeouts better * Add support for developer .env files * Eth1 improvements; Mainnet genesis state Notable changes: * The deposits table have been removed from the database. The client will no longer process all deposits on start-up. * The network metadata now includes a "state snapshot" of the deposit contract. This allows the client to skip syncing deposits made prior to the snapshot (i.e. genesis). Suitable metadata added for Pyrmont and Mainnet. * The Eth1 monitor won't be started unless there are validators attached to the node. * The genesis detection code is now optional and disabled by default * Bugfix: The client should not produce blocks that will fail validation when it hasn't downloaded the latest deposits yet * Bugfix: Work around the database corruption affecting Pyrmont nodes * Remove metadata for Toledo and Medalla
2020-11-24 21:21:47 +00:00
blockNumber = lastBlock.number,
Minor post-merge cleanups (#3945) https://github.com/status-im/nimbus-eth2/pull/3944 The use of nested `awaitWithRetries` calls would have resulted in an unexpected number of retries (3x3). We now use regular `await` in outer layer to avoid the problem. https://github.com/status-im/nimbus-eth2/pull/3943 The new code has an invariant that the `headMerkleizer` field in the `Eth1Chain` is always kept in sync with the blocks stored in the chain. This invariant is now enforced better by doing the necessary merkleizer updates in the `Eth1Chain.addBlock` function, in the `Eth1Chain.init` function and in the `Eth1Chain.reset` function.
2022-08-10 12:31:10 +00:00
depositsProcessed = lastBlock.depositCount
A hacky work-around for a web3 issue that may cause the client to go into a loop of failing requests
2020-11-17 19:50:07 +00:00
Add polling support in the Eth1Monitor (extracted from the merge branch)
2021-11-25 16:51:51 +00:00
func init(T: type FullBlockId, blk: Eth1BlockHeader|BlockObject): T =
FullBlockId(number: Eth1BlockNumber blk.number, hash: blk.hash)
Support for driving multiple EL nodes from a single Nimbus BN (#4465) * Support for driving multiple EL nodes from a single Nimbus BN Full list of changes: * Eth1Monitor has been renamed to ELManager to match its current responsibilities better. * The ELManager is no longer optional in the code (it won't have a nil value under any circumstances). * The support for subscribing for headers was removed as it only worked with WebSockets and contributed significant complexity while bringing only a very minor advantage. * The `--web3-url` parameter has been deprecated in favor of a new `--el` parameter. The new parameter has a reasonable default value and supports specifying a different JWT for each connection. Each connection can also be configured with a different set of responsibilities (e.g. download deposits, validate blocks and/or produce blocks). On the command-line, these properties can be configured through URL properties stored in the #anchor part of the URL. In TOML files, they come with a very natural syntax (althrough the URL scheme is also supported). * The previously scattered EL-related state and logic is now moved to `eth1_monitor.nim` (this module will be renamed to `el_manager.nim` in a follow-up commit). State is assigned properly either to the `ELManager` or the to individual `ELConnection` objects where appropriate. The ELManager executes all Engine API requests against all attached EL nodes, in parallel. It compares their results and if there is a disagreement regarding the validity of a certain payload, this is detected and the beacon node is protected from publishing a block with a potential execution layer consensus bug in it. The BN provides metrics per EL node for the number of successful or failed requests for each type Engine API requests. If an EL node goes offline and connectivity is resoted later, we report the problem and the remedy in edge-triggered fashion. * More progress towards implementing Deneb block production in the VC and comparing the value of blocks produced by the EL and the builder API. * Adds a Makefile target for the zhejiang testnet
2023-03-05 01:40:21 +00:00
func isNewLastBlock(m: ELManager, blk: Eth1BlockHeader|BlockObject): bool =
fix TTD before bellatrix (#4137) When TTD hits before Bellatrix, avoid waiting for new blocks and detect the TTD block as the terminal block hash even before Bellatrix hits. Also allow detecting EL genesis block as merge transition block. This fixes the local testnet simulation with Geth to actually merge.
2022-09-18 05:45:51 +00:00
m.latestEth1Block.isNone or blk.number.uint64 > m.latestEth1BlockNumber
import EL deposits even when EL is stuck (#3956) * import EL deposits even when EL is stuck The `eth1_monitor` only starts importing deposits once the EL reports a new head block. However, the EL may be stuck at a block, e.g., the TTD. By polling the latest EL block once after subscribing to new EL block events it is ensured that deposits are still imported in this situation. * also poll once on re-connects * update `eth1_latest_head` metric in poll mode * add comment about similar polling vs events parts * replace check with assert * `isNewLastBlock` helper
2022-08-12 19:44:55 +00:00
low attestations during epoch should instafail in CI; dbg -> warn level in `newPayload` log (#4830) * low attestations during epoch should instafail in CI; dbg -> warn level on newPayload log * improve newPayload warning message when no valid EL connected * reduce potential spam; make log spelling more consistent; use fatal/quit
2023-04-19 19:42:30 +00:00
func hasConnection*(m: ELManager): bool =
m.elConnections.len > 0
add `el_offline` to `/eth/v1/node/syncing` (#4860) Add compatibility with https://github.com/ethereum/beacon-APIs/pull/290 to the beacon node. Behaviour when configured with multiple ELs is not specified; intention suggests to indicate whether all ELs are offline.
2023-04-27 08:47:38 +00:00
func hasAnyWorkingConnection*(m: ELManager): bool =
allow `NeverTested` > `Working` transition for EL connection (#4991) Since #4960, the EL connection status can no longer transition from `NeverTested` to `Working`. Fix that, and also consider `NeverTested` connections as online for the purpose of the `el_offline` REST response.
2023-05-25 07:39:47 +00:00
m.elConnections.anyIt(it.state == Working or it.state == NeverTested)
add `el_offline` to `/eth/v1/node/syncing` (#4860) Add compatibility with https://github.com/ethereum/beacon-APIs/pull/290 to the beacon node. Behaviour when configured with multiple ELs is not specified; intention suggests to indicate whether all ELs are offline.
2023-04-27 08:47:38 +00:00
Support for driving multiple EL nodes from a single Nimbus BN (#4465) * Support for driving multiple EL nodes from a single Nimbus BN Full list of changes: * Eth1Monitor has been renamed to ELManager to match its current responsibilities better. * The ELManager is no longer optional in the code (it won't have a nil value under any circumstances). * The support for subscribing for headers was removed as it only worked with WebSockets and contributed significant complexity while bringing only a very minor advantage. * The `--web3-url` parameter has been deprecated in favor of a new `--el` parameter. The new parameter has a reasonable default value and supports specifying a different JWT for each connection. Each connection can also be configured with a different set of responsibilities (e.g. download deposits, validate blocks and/or produce blocks). On the command-line, these properties can be configured through URL properties stored in the #anchor part of the URL. In TOML files, they come with a very natural syntax (althrough the URL scheme is also supported). * The previously scattered EL-related state and logic is now moved to `eth1_monitor.nim` (this module will be renamed to `el_manager.nim` in a follow-up commit). State is assigned properly either to the `ELManager` or the to individual `ELConnection` objects where appropriate. The ELManager executes all Engine API requests against all attached EL nodes, in parallel. It compares their results and if there is a disagreement regarding the validity of a certain payload, this is detected and the beacon node is protected from publishing a block with a potential execution layer consensus bug in it. The BN provides metrics per EL node for the number of successful or failed requests for each type Engine API requests. If an EL node goes offline and connectivity is resoted later, we report the problem and the remedy in edge-triggered fashion. * More progress towards implementing Deneb block production in the VC and comparing the value of blocks produced by the EL and the builder API. * Adds a Makefile target for the zhejiang testnet
2023-03-05 01:40:21 +00:00
func hasProperlyConfiguredConnection*(m: ELManager): bool =
for connection in m.elConnections:
if connection.etcStatus == EtcStatus.match:
return true
Eth1 monitor fixes * Fix a resource leak introduced in https://github.com/status-im/nimbus-eth2/pull/3279 * Don't restart the Eth1 syncing proggress from scratch in case of monitor failures during Eth2 syncing. * Switch to the primary operator as soon as it is back online. * Log the web3 credentials in fewer places Other changes: The 'web3 test' command has been enhanced to obtain and print more data regarding the selected provider.
2022-01-31 17:28:26 +00:00
low attestations during epoch should instafail in CI; dbg -> warn level in `newPayload` log (#4830) * low attestations during epoch should instafail in CI; dbg -> warn level on newPayload log * improve newPayload warning message when no valid EL connected * reduce potential spam; make log spelling more consistent; use fatal/quit
2023-04-19 19:42:30 +00:00
false
Hotfix: Fix a race condition leading to a busy loop preventing progress in Eth1 syncing
2022-02-15 19:10:04 +00:00
Support for driving multiple EL nodes from a single Nimbus BN (#4465) * Support for driving multiple EL nodes from a single Nimbus BN Full list of changes: * Eth1Monitor has been renamed to ELManager to match its current responsibilities better. * The ELManager is no longer optional in the code (it won't have a nil value under any circumstances). * The support for subscribing for headers was removed as it only worked with WebSockets and contributed significant complexity while bringing only a very minor advantage. * The `--web3-url` parameter has been deprecated in favor of a new `--el` parameter. The new parameter has a reasonable default value and supports specifying a different JWT for each connection. Each connection can also be configured with a different set of responsibilities (e.g. download deposits, validate blocks and/or produce blocks). On the command-line, these properties can be configured through URL properties stored in the #anchor part of the URL. In TOML files, they come with a very natural syntax (althrough the URL scheme is also supported). * The previously scattered EL-related state and logic is now moved to `eth1_monitor.nim` (this module will be renamed to `el_manager.nim` in a follow-up commit). State is assigned properly either to the `ELManager` or the to individual `ELConnection` objects where appropriate. The ELManager executes all Engine API requests against all attached EL nodes, in parallel. It compares their results and if there is a disagreement regarding the validity of a certain payload, this is detected and the beacon node is protected from publishing a block with a potential execution layer consensus bug in it. The BN provides metrics per EL node for the number of successful or failed requests for each type Engine API requests. If an EL node goes offline and connectivity is resoted later, we report the problem and the remedy in edge-triggered fashion. * More progress towards implementing Deneb block production in the VC and comparing the value of blocks produced by the EL and the builder API. * Adds a Makefile target for the zhejiang testnet
2023-03-05 01:40:21 +00:00
proc startExchangeTransitionConfigurationLoop(m: ELManager) {.async.} =
debug "Starting exchange transition configuration loop"
More complete reset of the web3 provider on each failure; Fix #2184
2020-12-15 21:59:29 +00:00
Support for driving multiple EL nodes from a single Nimbus BN (#4465) * Support for driving multiple EL nodes from a single Nimbus BN Full list of changes: * Eth1Monitor has been renamed to ELManager to match its current responsibilities better. * The ELManager is no longer optional in the code (it won't have a nil value under any circumstances). * The support for subscribing for headers was removed as it only worked with WebSockets and contributed significant complexity while bringing only a very minor advantage. * The `--web3-url` parameter has been deprecated in favor of a new `--el` parameter. The new parameter has a reasonable default value and supports specifying a different JWT for each connection. Each connection can also be configured with a different set of responsibilities (e.g. download deposits, validate blocks and/or produce blocks). On the command-line, these properties can be configured through URL properties stored in the #anchor part of the URL. In TOML files, they come with a very natural syntax (althrough the URL scheme is also supported). * The previously scattered EL-related state and logic is now moved to `eth1_monitor.nim` (this module will be renamed to `el_manager.nim` in a follow-up commit). State is assigned properly either to the `ELManager` or the to individual `ELConnection` objects where appropriate. The ELManager executes all Engine API requests against all attached EL nodes, in parallel. It compares their results and if there is a disagreement regarding the validity of a certain payload, this is detected and the beacon node is protected from publishing a block with a potential execution layer consensus bug in it. The BN provides metrics per EL node for the number of successful or failed requests for each type Engine API requests. If an EL node goes offline and connectivity is resoted later, we report the problem and the remedy in edge-triggered fashion. * More progress towards implementing Deneb block production in the VC and comparing the value of blocks produced by the EL and the builder API. * Adds a Makefile target for the zhejiang testnet
2023-03-05 01:40:21 +00:00
while true:
update engine API URLs from v1.0.0-beta.2 to beta.3 (#4828)
2023-04-17 20:11:28 +00:00
# https://github.com/ethereum/execution-apis/blob/v1.0.0-beta.3/src/engine/paris.md#specification-3
Support for driving multiple EL nodes from a single Nimbus BN (#4465) * Support for driving multiple EL nodes from a single Nimbus BN Full list of changes: * Eth1Monitor has been renamed to ELManager to match its current responsibilities better. * The ELManager is no longer optional in the code (it won't have a nil value under any circumstances). * The support for subscribing for headers was removed as it only worked with WebSockets and contributed significant complexity while bringing only a very minor advantage. * The `--web3-url` parameter has been deprecated in favor of a new `--el` parameter. The new parameter has a reasonable default value and supports specifying a different JWT for each connection. Each connection can also be configured with a different set of responsibilities (e.g. download deposits, validate blocks and/or produce blocks). On the command-line, these properties can be configured through URL properties stored in the #anchor part of the URL. In TOML files, they come with a very natural syntax (althrough the URL scheme is also supported). * The previously scattered EL-related state and logic is now moved to `eth1_monitor.nim` (this module will be renamed to `el_manager.nim` in a follow-up commit). State is assigned properly either to the `ELManager` or the to individual `ELConnection` objects where appropriate. The ELManager executes all Engine API requests against all attached EL nodes, in parallel. It compares their results and if there is a disagreement regarding the validity of a certain payload, this is detected and the beacon node is protected from publishing a block with a potential execution layer consensus bug in it. The BN provides metrics per EL node for the number of successful or failed requests for each type Engine API requests. If an EL node goes offline and connectivity is resoted later, we report the problem and the remedy in edge-triggered fashion. * More progress towards implementing Deneb block production in the VC and comparing the value of blocks produced by the EL and the builder API. * Adds a Makefile target for the zhejiang testnet
2023-03-05 01:40:21 +00:00
debug "Exchange transition configuration tick"
traceAsyncErrors m.exchangeTransitionConfiguration()
More metrics for the EL monitor (#4707) * `engine_api_response_time` provides a histogram for the Engine API response times for each unique pair ot URL and request type. * All engine API requests are now tracked Other changes: The client will no longer exit on start-up if it fails to connect to a properly configured EL node.
2023-03-09 17:29:04 +00:00
await sleepAsync(60.seconds)
Support for driving multiple EL nodes from a single Nimbus BN (#4465) * Support for driving multiple EL nodes from a single Nimbus BN Full list of changes: * Eth1Monitor has been renamed to ELManager to match its current responsibilities better. * The ELManager is no longer optional in the code (it won't have a nil value under any circumstances). * The support for subscribing for headers was removed as it only worked with WebSockets and contributed significant complexity while bringing only a very minor advantage. * The `--web3-url` parameter has been deprecated in favor of a new `--el` parameter. The new parameter has a reasonable default value and supports specifying a different JWT for each connection. Each connection can also be configured with a different set of responsibilities (e.g. download deposits, validate blocks and/or produce blocks). On the command-line, these properties can be configured through URL properties stored in the #anchor part of the URL. In TOML files, they come with a very natural syntax (althrough the URL scheme is also supported). * The previously scattered EL-related state and logic is now moved to `eth1_monitor.nim` (this module will be renamed to `el_manager.nim` in a follow-up commit). State is assigned properly either to the `ELManager` or the to individual `ELConnection` objects where appropriate. The ELManager executes all Engine API requests against all attached EL nodes, in parallel. It compares their results and if there is a disagreement regarding the validity of a certain payload, this is detected and the beacon node is protected from publishing a block with a potential execution layer consensus bug in it. The BN provides metrics per EL node for the number of successful or failed requests for each type Engine API requests. If an EL node goes offline and connectivity is resoted later, we report the problem and the remedy in edge-triggered fashion. * More progress towards implementing Deneb block production in the VC and comparing the value of blocks produced by the EL and the builder API. * Adds a Makefile target for the zhejiang testnet
2023-03-05 01:40:21 +00:00
proc syncEth1Chain(m: ELManager, connection: ELConnection) {.async.} =
More forceful EL connection resetting (fixes #4424) (#4708)
2023-03-09 23:41:28 +00:00
let rpcClient = awaitOrRaiseOnTimeout(connection.connectedRpcClient(),
1.seconds)
Support for driving multiple EL nodes from a single Nimbus BN (#4465) * Support for driving multiple EL nodes from a single Nimbus BN Full list of changes: * Eth1Monitor has been renamed to ELManager to match its current responsibilities better. * The ELManager is no longer optional in the code (it won't have a nil value under any circumstances). * The support for subscribing for headers was removed as it only worked with WebSockets and contributed significant complexity while bringing only a very minor advantage. * The `--web3-url` parameter has been deprecated in favor of a new `--el` parameter. The new parameter has a reasonable default value and supports specifying a different JWT for each connection. Each connection can also be configured with a different set of responsibilities (e.g. download deposits, validate blocks and/or produce blocks). On the command-line, these properties can be configured through URL properties stored in the #anchor part of the URL. In TOML files, they come with a very natural syntax (althrough the URL scheme is also supported). * The previously scattered EL-related state and logic is now moved to `eth1_monitor.nim` (this module will be renamed to `el_manager.nim` in a follow-up commit). State is assigned properly either to the `ELManager` or the to individual `ELConnection` objects where appropriate. The ELManager executes all Engine API requests against all attached EL nodes, in parallel. It compares their results and if there is a disagreement regarding the validity of a certain payload, this is detected and the beacon node is protected from publishing a block with a potential execution layer consensus bug in it. The BN provides metrics per EL node for the number of successful or failed requests for each type Engine API requests. If an EL node goes offline and connectivity is resoted later, we report the problem and the remedy in edge-triggered fashion. * More progress towards implementing Deneb block production in the VC and comparing the value of blocks produced by the EL and the builder API. * Adds a Makefile target for the zhejiang testnet
2023-03-05 01:40:21 +00:00
let
Prevent a potential panic on dereferencing a 'none' value
2023-03-14 16:05:21 +00:00
# BEWARE
# `connectedRpcClient` guarantees that connection.web3 will not be
# `none` here, but it's not safe to initialize this later (e.g closer
# to where it's used) because `connection.web3` may be set to `none`
# at any time after a failed request. Luckily, the `contractSender`
# object is very cheap to create.
depositContract = connection.web3.get.contractSender(
DepositContract, m.depositContractAddress)
Support for driving multiple EL nodes from a single Nimbus BN (#4465) * Support for driving multiple EL nodes from a single Nimbus BN Full list of changes: * Eth1Monitor has been renamed to ELManager to match its current responsibilities better. * The ELManager is no longer optional in the code (it won't have a nil value under any circumstances). * The support for subscribing for headers was removed as it only worked with WebSockets and contributed significant complexity while bringing only a very minor advantage. * The `--web3-url` parameter has been deprecated in favor of a new `--el` parameter. The new parameter has a reasonable default value and supports specifying a different JWT for each connection. Each connection can also be configured with a different set of responsibilities (e.g. download deposits, validate blocks and/or produce blocks). On the command-line, these properties can be configured through URL properties stored in the #anchor part of the URL. In TOML files, they come with a very natural syntax (althrough the URL scheme is also supported). * The previously scattered EL-related state and logic is now moved to `eth1_monitor.nim` (this module will be renamed to `el_manager.nim` in a follow-up commit). State is assigned properly either to the `ELManager` or the to individual `ELConnection` objects where appropriate. The ELManager executes all Engine API requests against all attached EL nodes, in parallel. It compares their results and if there is a disagreement regarding the validity of a certain payload, this is detected and the beacon node is protected from publishing a block with a potential execution layer consensus bug in it. The BN provides metrics per EL node for the number of successful or failed requests for each type Engine API requests. If an EL node goes offline and connectivity is resoted later, we report the problem and the remedy in edge-triggered fashion. * More progress towards implementing Deneb block production in the VC and comparing the value of blocks produced by the EL and the builder API. * Adds a Makefile target for the zhejiang testnet
2023-03-05 01:40:21 +00:00
shouldProcessDeposits = not (
m.depositContractAddress.isZeroMemory or
m.eth1Chain.finalizedBlockHash.data.isZeroMemory)
trace "Starting syncEth1Chain", shouldProcessDeposits
Eth1 monitor fixes * Fix a resource leak introduced in https://github.com/status-im/nimbus-eth2/pull/3279 * Don't restart the Eth1 syncing proggress from scratch in case of monitor failures during Eth2 syncing. * Switch to the primary operator as soon as it is back online. * Log the web3 credentials in fewer places Other changes: The 'web3 test' command has been enhanced to obtain and print more data regarding the selected provider.
2022-01-31 17:28:26 +00:00
Support for driving multiple EL nodes from a single Nimbus BN (#4465) * Support for driving multiple EL nodes from a single Nimbus BN Full list of changes: * Eth1Monitor has been renamed to ELManager to match its current responsibilities better. * The ELManager is no longer optional in the code (it won't have a nil value under any circumstances). * The support for subscribing for headers was removed as it only worked with WebSockets and contributed significant complexity while bringing only a very minor advantage. * The `--web3-url` parameter has been deprecated in favor of a new `--el` parameter. The new parameter has a reasonable default value and supports specifying a different JWT for each connection. Each connection can also be configured with a different set of responsibilities (e.g. download deposits, validate blocks and/or produce blocks). On the command-line, these properties can be configured through URL properties stored in the #anchor part of the URL. In TOML files, they come with a very natural syntax (althrough the URL scheme is also supported). * The previously scattered EL-related state and logic is now moved to `eth1_monitor.nim` (this module will be renamed to `el_manager.nim` in a follow-up commit). State is assigned properly either to the `ELManager` or the to individual `ELConnection` objects where appropriate. The ELManager executes all Engine API requests against all attached EL nodes, in parallel. It compares their results and if there is a disagreement regarding the validity of a certain payload, this is detected and the beacon node is protected from publishing a block with a potential execution layer consensus bug in it. The BN provides metrics per EL node for the number of successful or failed requests for each type Engine API requests. If an EL node goes offline and connectivity is resoted later, we report the problem and the remedy in edge-triggered fashion. * More progress towards implementing Deneb block production in the VC and comparing the value of blocks produced by the EL and the builder API. * Adds a Makefile target for the zhejiang testnet
2023-03-05 01:40:21 +00:00
logScope:
url = connection.engineUrl.url
Eth1 monitor fixes * Fix a resource leak introduced in https://github.com/status-im/nimbus-eth2/pull/3279 * Don't restart the Eth1 syncing proggress from scratch in case of monitor failures during Eth2 syncing. * Switch to the primary operator as soon as it is back online. * Log the web3 credentials in fewer places Other changes: The 'web3 test' command has been enhanced to obtain and print more data regarding the selected provider.
2022-01-31 17:28:26 +00:00
# We might need to reset the chain if the new provider disagrees
# with the previous one regarding the history of the chain or if
# we have detected a conensus violation - our view disagreeing with
# the majority of the validators in the network.
#
# Consensus violations happen in practice because the web3 providers
# sometimes return incomplete or incorrect deposit log events even
# when they don't indicate any errors in the response. When this
# happens, we are usually able to download the data successfully
# on the second attempt.
Support for driving multiple EL nodes from a single Nimbus BN (#4465) * Support for driving multiple EL nodes from a single Nimbus BN Full list of changes: * Eth1Monitor has been renamed to ELManager to match its current responsibilities better. * The ELManager is no longer optional in the code (it won't have a nil value under any circumstances). * The support for subscribing for headers was removed as it only worked with WebSockets and contributed significant complexity while bringing only a very minor advantage. * The `--web3-url` parameter has been deprecated in favor of a new `--el` parameter. The new parameter has a reasonable default value and supports specifying a different JWT for each connection. Each connection can also be configured with a different set of responsibilities (e.g. download deposits, validate blocks and/or produce blocks). On the command-line, these properties can be configured through URL properties stored in the #anchor part of the URL. In TOML files, they come with a very natural syntax (althrough the URL scheme is also supported). * The previously scattered EL-related state and logic is now moved to `eth1_monitor.nim` (this module will be renamed to `el_manager.nim` in a follow-up commit). State is assigned properly either to the `ELManager` or the to individual `ELConnection` objects where appropriate. The ELManager executes all Engine API requests against all attached EL nodes, in parallel. It compares their results and if there is a disagreement regarding the validity of a certain payload, this is detected and the beacon node is protected from publishing a block with a potential execution layer consensus bug in it. The BN provides metrics per EL node for the number of successful or failed requests for each type Engine API requests. If an EL node goes offline and connectivity is resoted later, we report the problem and the remedy in edge-triggered fashion. * More progress towards implementing Deneb block production in the VC and comparing the value of blocks produced by the EL and the builder API. * Adds a Makefile target for the zhejiang testnet
2023-03-05 01:40:21 +00:00
#
# TODO
# Perhaps the above problem was manifesting only with the obsolete
# JSON-RPC data providers, which can no longer be used with Nimbus.
if m.eth1Chain.blocks.len > 0:
let needsReset = m.eth1Chain.hasConsensusViolation or (block:
Eth1 monitor fixes * Fix a resource leak introduced in https://github.com/status-im/nimbus-eth2/pull/3279 * Don't restart the Eth1 syncing proggress from scratch in case of monitor failures during Eth2 syncing. * Switch to the primary operator as soon as it is back online. * Log the web3 credentials in fewer places Other changes: The 'web3 test' command has been enhanced to obtain and print more data regarding the selected provider.
2022-01-31 17:28:26 +00:00
let
Support for driving multiple EL nodes from a single Nimbus BN (#4465) * Support for driving multiple EL nodes from a single Nimbus BN Full list of changes: * Eth1Monitor has been renamed to ELManager to match its current responsibilities better. * The ELManager is no longer optional in the code (it won't have a nil value under any circumstances). * The support for subscribing for headers was removed as it only worked with WebSockets and contributed significant complexity while bringing only a very minor advantage. * The `--web3-url` parameter has been deprecated in favor of a new `--el` parameter. The new parameter has a reasonable default value and supports specifying a different JWT for each connection. Each connection can also be configured with a different set of responsibilities (e.g. download deposits, validate blocks and/or produce blocks). On the command-line, these properties can be configured through URL properties stored in the #anchor part of the URL. In TOML files, they come with a very natural syntax (althrough the URL scheme is also supported). * The previously scattered EL-related state and logic is now moved to `eth1_monitor.nim` (this module will be renamed to `el_manager.nim` in a follow-up commit). State is assigned properly either to the `ELManager` or the to individual `ELConnection` objects where appropriate. The ELManager executes all Engine API requests against all attached EL nodes, in parallel. It compares their results and if there is a disagreement regarding the validity of a certain payload, this is detected and the beacon node is protected from publishing a block with a potential execution layer consensus bug in it. The BN provides metrics per EL node for the number of successful or failed requests for each type Engine API requests. If an EL node goes offline and connectivity is resoted later, we report the problem and the remedy in edge-triggered fashion. * More progress towards implementing Deneb block production in the VC and comparing the value of blocks produced by the EL and the builder API. * Adds a Makefile target for the zhejiang testnet
2023-03-05 01:40:21 +00:00
lastKnownBlock = m.eth1Chain.blocks.peekLast
More metrics for the EL monitor (#4707) * `engine_api_response_time` provides a histogram for the Engine API response times for each unique pair ot URL and request type. * All engine API requests are now tracked Other changes: The client will no longer exit on start-up if it fails to connect to a properly configured EL node.
2023-03-09 17:29:04 +00:00
matchingBlockAtNewProvider = connection.trackedRequestWithTimeout(
"getBlockByNumber",
Support for driving multiple EL nodes from a single Nimbus BN (#4465) * Support for driving multiple EL nodes from a single Nimbus BN Full list of changes: * Eth1Monitor has been renamed to ELManager to match its current responsibilities better. * The ELManager is no longer optional in the code (it won't have a nil value under any circumstances). * The support for subscribing for headers was removed as it only worked with WebSockets and contributed significant complexity while bringing only a very minor advantage. * The `--web3-url` parameter has been deprecated in favor of a new `--el` parameter. The new parameter has a reasonable default value and supports specifying a different JWT for each connection. Each connection can also be configured with a different set of responsibilities (e.g. download deposits, validate blocks and/or produce blocks). On the command-line, these properties can be configured through URL properties stored in the #anchor part of the URL. In TOML files, they come with a very natural syntax (althrough the URL scheme is also supported). * The previously scattered EL-related state and logic is now moved to `eth1_monitor.nim` (this module will be renamed to `el_manager.nim` in a follow-up commit). State is assigned properly either to the `ELManager` or the to individual `ELConnection` objects where appropriate. The ELManager executes all Engine API requests against all attached EL nodes, in parallel. It compares their results and if there is a disagreement regarding the validity of a certain payload, this is detected and the beacon node is protected from publishing a block with a potential execution layer consensus bug in it. The BN provides metrics per EL node for the number of successful or failed requests for each type Engine API requests. If an EL node goes offline and connectivity is resoted later, we report the problem and the remedy in edge-triggered fashion. * More progress towards implementing Deneb block production in the VC and comparing the value of blocks produced by the EL and the builder API. * Adds a Makefile target for the zhejiang testnet
2023-03-05 01:40:21 +00:00
rpcClient.getBlockByNumber(lastKnownBlock.number),
web3RequestsTimeout)
Eth1 monitor fixes * Fix a resource leak introduced in https://github.com/status-im/nimbus-eth2/pull/3279 * Don't restart the Eth1 syncing proggress from scratch in case of monitor failures during Eth2 syncing. * Switch to the primary operator as soon as it is back online. * Log the web3 credentials in fewer places Other changes: The 'web3 test' command has been enhanced to obtain and print more data regarding the selected provider.
2022-01-31 17:28:26 +00:00
Minor post-merge cleanups (#3945) https://github.com/status-im/nimbus-eth2/pull/3944 The use of nested `awaitWithRetries` calls would have resulted in an unexpected number of retries (3x3). We now use regular `await` in outer layer to avoid the problem. https://github.com/status-im/nimbus-eth2/pull/3943 The new code has an invariant that the `headMerkleizer` field in the `Eth1Chain` is always kept in sync with the blocks stored in the chain. This invariant is now enforced better by doing the necessary merkleizer updates in the `Eth1Chain.addBlock` function, in the `Eth1Chain.init` function and in the `Eth1Chain.reset` function.
2022-08-10 12:31:10 +00:00
lastKnownBlock.hash.asBlockHash != matchingBlockAtNewProvider.hash)
Eth1 monitor fixes * Fix a resource leak introduced in https://github.com/status-im/nimbus-eth2/pull/3279 * Don't restart the Eth1 syncing proggress from scratch in case of monitor failures during Eth2 syncing. * Switch to the primary operator as soon as it is back online. * Log the web3 credentials in fewer places Other changes: The 'web3 test' command has been enhanced to obtain and print more data regarding the selected provider.
2022-01-31 17:28:26 +00:00
if needsReset:
Support for driving multiple EL nodes from a single Nimbus BN (#4465) * Support for driving multiple EL nodes from a single Nimbus BN Full list of changes: * Eth1Monitor has been renamed to ELManager to match its current responsibilities better. * The ELManager is no longer optional in the code (it won't have a nil value under any circumstances). * The support for subscribing for headers was removed as it only worked with WebSockets and contributed significant complexity while bringing only a very minor advantage. * The `--web3-url` parameter has been deprecated in favor of a new `--el` parameter. The new parameter has a reasonable default value and supports specifying a different JWT for each connection. Each connection can also be configured with a different set of responsibilities (e.g. download deposits, validate blocks and/or produce blocks). On the command-line, these properties can be configured through URL properties stored in the #anchor part of the URL. In TOML files, they come with a very natural syntax (althrough the URL scheme is also supported). * The previously scattered EL-related state and logic is now moved to `eth1_monitor.nim` (this module will be renamed to `el_manager.nim` in a follow-up commit). State is assigned properly either to the `ELManager` or the to individual `ELConnection` objects where appropriate. The ELManager executes all Engine API requests against all attached EL nodes, in parallel. It compares their results and if there is a disagreement regarding the validity of a certain payload, this is detected and the beacon node is protected from publishing a block with a potential execution layer consensus bug in it. The BN provides metrics per EL node for the number of successful or failed requests for each type Engine API requests. If an EL node goes offline and connectivity is resoted later, we report the problem and the remedy in edge-triggered fashion. * More progress towards implementing Deneb block production in the VC and comparing the value of blocks produced by the EL and the builder API. * Adds a Makefile target for the zhejiang testnet
2023-03-05 01:40:21 +00:00
trace "Resetting the Eth1 chain",
hasConsensusViolation = m.eth1Chain.hasConsensusViolation
m.eth1Chain.clear()
Support for obtaining deposit snapshots during trustedNodeSync (#4303) Other changes: * More optimal search for TTD block. * Add timeouts to all REST requests during trusted node sync. Fixes #4037 * Removed support for storing a deposit snapshot in the network metadata.
2022-12-07 10:24:51 +00:00
Bellatrix TTD detection (#3745) * Bellatrix TTD detection * Update beacon_chain/eth1/eth1_monitor.nim Co-authored-by: Etan Kissling <etan@status.im> * Update beacon_chain/nimbus_beacon_node.nim Co-authored-by: tersec <tersec@users.noreply.github.com> Co-authored-by: Etan Kissling <etan@status.im> Co-authored-by: tersec <tersec@users.noreply.github.com>
2022-06-15 02:38:27 +00:00
var eth1SyncedTo: Eth1BlockNumber
resume EL sync at correct block on reconnect (#3957) When connection to the EL is lost as part of EL deposits importing, the targeted block range to sync would reset. This is changed to properly remember import progress across reconnects.
2022-08-12 13:48:33 +00:00
if shouldProcessDeposits:
Support for driving multiple EL nodes from a single Nimbus BN (#4465) * Support for driving multiple EL nodes from a single Nimbus BN Full list of changes: * Eth1Monitor has been renamed to ELManager to match its current responsibilities better. * The ELManager is no longer optional in the code (it won't have a nil value under any circumstances). * The support for subscribing for headers was removed as it only worked with WebSockets and contributed significant complexity while bringing only a very minor advantage. * The `--web3-url` parameter has been deprecated in favor of a new `--el` parameter. The new parameter has a reasonable default value and supports specifying a different JWT for each connection. Each connection can also be configured with a different set of responsibilities (e.g. download deposits, validate blocks and/or produce blocks). On the command-line, these properties can be configured through URL properties stored in the #anchor part of the URL. In TOML files, they come with a very natural syntax (althrough the URL scheme is also supported). * The previously scattered EL-related state and logic is now moved to `eth1_monitor.nim` (this module will be renamed to `el_manager.nim` in a follow-up commit). State is assigned properly either to the `ELManager` or the to individual `ELConnection` objects where appropriate. The ELManager executes all Engine API requests against all attached EL nodes, in parallel. It compares their results and if there is a disagreement regarding the validity of a certain payload, this is detected and the beacon node is protected from publishing a block with a potential execution layer consensus bug in it. The BN provides metrics per EL node for the number of successful or failed requests for each type Engine API requests. If an EL node goes offline and connectivity is resoted later, we report the problem and the remedy in edge-triggered fashion. * More progress towards implementing Deneb block production in the VC and comparing the value of blocks produced by the EL and the builder API. * Adds a Makefile target for the zhejiang testnet
2023-03-05 01:40:21 +00:00
if m.eth1Chain.blocks.len == 0:
let finalizedBlockHash = m.eth1Chain.finalizedBlockHash.asBlockHash
let startBlock =
More metrics for the EL monitor (#4707) * `engine_api_response_time` provides a histogram for the Engine API response times for each unique pair ot URL and request type. * All engine API requests are now tracked Other changes: The client will no longer exit on start-up if it fails to connect to a properly configured EL node.
2023-03-09 17:29:04 +00:00
connection.trackedRequestWithTimeout(
"getBlockByHash",
rpcClient.getBlockByHash(finalizedBlockHash),
web3RequestsTimeout)
Support for driving multiple EL nodes from a single Nimbus BN (#4465) * Support for driving multiple EL nodes from a single Nimbus BN Full list of changes: * Eth1Monitor has been renamed to ELManager to match its current responsibilities better. * The ELManager is no longer optional in the code (it won't have a nil value under any circumstances). * The support for subscribing for headers was removed as it only worked with WebSockets and contributed significant complexity while bringing only a very minor advantage. * The `--web3-url` parameter has been deprecated in favor of a new `--el` parameter. The new parameter has a reasonable default value and supports specifying a different JWT for each connection. Each connection can also be configured with a different set of responsibilities (e.g. download deposits, validate blocks and/or produce blocks). On the command-line, these properties can be configured through URL properties stored in the #anchor part of the URL. In TOML files, they come with a very natural syntax (althrough the URL scheme is also supported). * The previously scattered EL-related state and logic is now moved to `eth1_monitor.nim` (this module will be renamed to `el_manager.nim` in a follow-up commit). State is assigned properly either to the `ELManager` or the to individual `ELConnection` objects where appropriate. The ELManager executes all Engine API requests against all attached EL nodes, in parallel. It compares their results and if there is a disagreement regarding the validity of a certain payload, this is detected and the beacon node is protected from publishing a block with a potential execution layer consensus bug in it. The BN provides metrics per EL node for the number of successful or failed requests for each type Engine API requests. If an EL node goes offline and connectivity is resoted later, we report the problem and the remedy in edge-triggered fashion. * More progress towards implementing Deneb block production in the VC and comparing the value of blocks produced by the EL and the builder API. * Adds a Makefile target for the zhejiang testnet
2023-03-05 01:40:21 +00:00
m.eth1Chain.addBlock Eth1Block(
hash: m.eth1Chain.finalizedBlockHash,
resume EL sync at correct block on reconnect (#3957) When connection to the EL is lost as part of EL deposits importing, the targeted block range to sync would reset. This is changed to properly remember import progress across reconnects.
2022-08-12 13:48:33 +00:00
number: Eth1BlockNumber startBlock.number,
timestamp: Eth1BlockTimestamp startBlock.timestamp)
Tentative fix for the Pyrmont Eth1 voting issue
2020-11-19 17:19:03 +00:00
Support for driving multiple EL nodes from a single Nimbus BN (#4465) * Support for driving multiple EL nodes from a single Nimbus BN Full list of changes: * Eth1Monitor has been renamed to ELManager to match its current responsibilities better. * The ELManager is no longer optional in the code (it won't have a nil value under any circumstances). * The support for subscribing for headers was removed as it only worked with WebSockets and contributed significant complexity while bringing only a very minor advantage. * The `--web3-url` parameter has been deprecated in favor of a new `--el` parameter. The new parameter has a reasonable default value and supports specifying a different JWT for each connection. Each connection can also be configured with a different set of responsibilities (e.g. download deposits, validate blocks and/or produce blocks). On the command-line, these properties can be configured through URL properties stored in the #anchor part of the URL. In TOML files, they come with a very natural syntax (althrough the URL scheme is also supported). * The previously scattered EL-related state and logic is now moved to `eth1_monitor.nim` (this module will be renamed to `el_manager.nim` in a follow-up commit). State is assigned properly either to the `ELManager` or the to individual `ELConnection` objects where appropriate. The ELManager executes all Engine API requests against all attached EL nodes, in parallel. It compares their results and if there is a disagreement regarding the validity of a certain payload, this is detected and the beacon node is protected from publishing a block with a potential execution layer consensus bug in it. The BN provides metrics per EL node for the number of successful or failed requests for each type Engine API requests. If an EL node goes offline and connectivity is resoted later, we report the problem and the remedy in edge-triggered fashion. * More progress towards implementing Deneb block production in the VC and comparing the value of blocks produced by the EL and the builder API. * Adds a Makefile target for the zhejiang testnet
2023-03-05 01:40:21 +00:00
eth1SyncedTo = m.eth1Chain.blocks[^1].number
Bellatrix TTD detection (#3745) * Bellatrix TTD detection * Update beacon_chain/eth1/eth1_monitor.nim Co-authored-by: Etan Kissling <etan@status.im> * Update beacon_chain/nimbus_beacon_node.nim Co-authored-by: tersec <tersec@users.noreply.github.com> Co-authored-by: Etan Kissling <etan@status.im> Co-authored-by: tersec <tersec@users.noreply.github.com>
2022-06-15 02:38:27 +00:00
eth1_synced_head.set eth1SyncedTo.toGaugeValue
eth1_finalized_head.set eth1SyncedTo.toGaugeValue
eth1_finalized_deposits.set(
Support for driving multiple EL nodes from a single Nimbus BN (#4465) * Support for driving multiple EL nodes from a single Nimbus BN Full list of changes: * Eth1Monitor has been renamed to ELManager to match its current responsibilities better. * The ELManager is no longer optional in the code (it won't have a nil value under any circumstances). * The support for subscribing for headers was removed as it only worked with WebSockets and contributed significant complexity while bringing only a very minor advantage. * The `--web3-url` parameter has been deprecated in favor of a new `--el` parameter. The new parameter has a reasonable default value and supports specifying a different JWT for each connection. Each connection can also be configured with a different set of responsibilities (e.g. download deposits, validate blocks and/or produce blocks). On the command-line, these properties can be configured through URL properties stored in the #anchor part of the URL. In TOML files, they come with a very natural syntax (althrough the URL scheme is also supported). * The previously scattered EL-related state and logic is now moved to `eth1_monitor.nim` (this module will be renamed to `el_manager.nim` in a follow-up commit). State is assigned properly either to the `ELManager` or the to individual `ELConnection` objects where appropriate. The ELManager executes all Engine API requests against all attached EL nodes, in parallel. It compares their results and if there is a disagreement regarding the validity of a certain payload, this is detected and the beacon node is protected from publishing a block with a potential execution layer consensus bug in it. The BN provides metrics per EL node for the number of successful or failed requests for each type Engine API requests. If an EL node goes offline and connectivity is resoted later, we report the problem and the remedy in edge-triggered fashion. * More progress towards implementing Deneb block production in the VC and comparing the value of blocks produced by the EL and the builder API. * Adds a Makefile target for the zhejiang testnet
2023-03-05 01:40:21 +00:00
m.eth1Chain.finalizedDepositsMerkleizer.getChunkCount.toGaugeValue)
Add some useful Eth1 monitor metrics
2020-12-09 22:44:59 +00:00
Support for driving multiple EL nodes from a single Nimbus BN (#4465) * Support for driving multiple EL nodes from a single Nimbus BN Full list of changes: * Eth1Monitor has been renamed to ELManager to match its current responsibilities better. * The ELManager is no longer optional in the code (it won't have a nil value under any circumstances). * The support for subscribing for headers was removed as it only worked with WebSockets and contributed significant complexity while bringing only a very minor advantage. * The `--web3-url` parameter has been deprecated in favor of a new `--el` parameter. The new parameter has a reasonable default value and supports specifying a different JWT for each connection. Each connection can also be configured with a different set of responsibilities (e.g. download deposits, validate blocks and/or produce blocks). On the command-line, these properties can be configured through URL properties stored in the #anchor part of the URL. In TOML files, they come with a very natural syntax (althrough the URL scheme is also supported). * The previously scattered EL-related state and logic is now moved to `eth1_monitor.nim` (this module will be renamed to `el_manager.nim` in a follow-up commit). State is assigned properly either to the `ELManager` or the to individual `ELConnection` objects where appropriate. The ELManager executes all Engine API requests against all attached EL nodes, in parallel. It compares their results and if there is a disagreement regarding the validity of a certain payload, this is detected and the beacon node is protected from publishing a block with a potential execution layer consensus bug in it. The BN provides metrics per EL node for the number of successful or failed requests for each type Engine API requests. If an EL node goes offline and connectivity is resoted later, we report the problem and the remedy in edge-triggered fashion. * More progress towards implementing Deneb block production in the VC and comparing the value of blocks produced by the EL and the builder API. * Adds a Makefile target for the zhejiang testnet
2023-03-05 01:40:21 +00:00
debug "Starting Eth1 syncing", `from` = shortLog(m.eth1Chain.blocks[^1])
Less risk for Eth1-induced delays in block proposal
2020-11-30 23:59:35 +00:00
import EL deposits even when EL is stuck (#3956) * import EL deposits even when EL is stuck The `eth1_monitor` only starts importing deposits once the EL reports a new head block. However, the EL may be stuck at a block, e.g., the TTD. By polling the latest EL block once after subscribing to new EL block events it is ensured that deposits are still imported in this situation. * also poll once on re-connects * update `eth1_latest_head` metric in poll mode * add comment about similar polling vs events parts * replace check with assert * `isNewLastBlock` helper
2022-08-12 19:44:55 +00:00
var didPollOnce = false
Further reduce the number of RPC requests; Significantly faster ETH1 syncing
2020-11-03 01:21:07 +00:00
while true:
Support for driving multiple EL nodes from a single Nimbus BN (#4465) * Support for driving multiple EL nodes from a single Nimbus BN Full list of changes: * Eth1Monitor has been renamed to ELManager to match its current responsibilities better. * The ELManager is no longer optional in the code (it won't have a nil value under any circumstances). * The support for subscribing for headers was removed as it only worked with WebSockets and contributed significant complexity while bringing only a very minor advantage. * The `--web3-url` parameter has been deprecated in favor of a new `--el` parameter. The new parameter has a reasonable default value and supports specifying a different JWT for each connection. Each connection can also be configured with a different set of responsibilities (e.g. download deposits, validate blocks and/or produce blocks). On the command-line, these properties can be configured through URL properties stored in the #anchor part of the URL. In TOML files, they come with a very natural syntax (althrough the URL scheme is also supported). * The previously scattered EL-related state and logic is now moved to `eth1_monitor.nim` (this module will be renamed to `el_manager.nim` in a follow-up commit). State is assigned properly either to the `ELManager` or the to individual `ELConnection` objects where appropriate. The ELManager executes all Engine API requests against all attached EL nodes, in parallel. It compares their results and if there is a disagreement regarding the validity of a certain payload, this is detected and the beacon node is protected from publishing a block with a potential execution layer consensus bug in it. The BN provides metrics per EL node for the number of successful or failed requests for each type Engine API requests. If an EL node goes offline and connectivity is resoted later, we report the problem and the remedy in edge-triggered fashion. * More progress towards implementing Deneb block production in the VC and comparing the value of blocks produced by the EL and the builder API. * Adds a Makefile target for the zhejiang testnet
2023-03-05 01:40:21 +00:00
debug "syncEth1Chain tick"
Further reduce the number of RPC requests; Significantly faster ETH1 syncing
2020-11-03 01:21:07 +00:00
if bnStatus == BeaconNodeStatus.Stopping:
More complete reset of the web3 provider on each failure; Fix #2184
2020-12-15 21:59:29 +00:00
await m.stop()
Further reduce the number of RPC requests; Significantly faster ETH1 syncing
2020-11-03 01:21:07 +00:00
return
Simplify the mainchain monitor
2020-10-14 14:04:08 +00:00
Support for driving multiple EL nodes from a single Nimbus BN (#4465) * Support for driving multiple EL nodes from a single Nimbus BN Full list of changes: * Eth1Monitor has been renamed to ELManager to match its current responsibilities better. * The ELManager is no longer optional in the code (it won't have a nil value under any circumstances). * The support for subscribing for headers was removed as it only worked with WebSockets and contributed significant complexity while bringing only a very minor advantage. * The `--web3-url` parameter has been deprecated in favor of a new `--el` parameter. The new parameter has a reasonable default value and supports specifying a different JWT for each connection. Each connection can also be configured with a different set of responsibilities (e.g. download deposits, validate blocks and/or produce blocks). On the command-line, these properties can be configured through URL properties stored in the #anchor part of the URL. In TOML files, they come with a very natural syntax (althrough the URL scheme is also supported). * The previously scattered EL-related state and logic is now moved to `eth1_monitor.nim` (this module will be renamed to `el_manager.nim` in a follow-up commit). State is assigned properly either to the `ELManager` or the to individual `ELConnection` objects where appropriate. The ELManager executes all Engine API requests against all attached EL nodes, in parallel. It compares their results and if there is a disagreement regarding the validity of a certain payload, this is detected and the beacon node is protected from publishing a block with a potential execution layer consensus bug in it. The BN provides metrics per EL node for the number of successful or failed requests for each type Engine API requests. If an EL node goes offline and connectivity is resoted later, we report the problem and the remedy in edge-triggered fashion. * More progress towards implementing Deneb block production in the VC and comparing the value of blocks produced by the EL and the builder API. * Adds a Makefile target for the zhejiang testnet
2023-03-05 01:40:21 +00:00
if m.eth1Chain.hasConsensusViolation:
Fix #2261 Also bumps Confutils to allow setting the hidden --web3-mode param (to allow testing the eth1 syncing without validators)
2021-01-29 21:21:44 +00:00
raise newException(CorruptDataProvider, "Eth1 chain contradicts Eth2 consensus")
Support for driving multiple EL nodes from a single Nimbus BN (#4465) * Support for driving multiple EL nodes from a single Nimbus BN Full list of changes: * Eth1Monitor has been renamed to ELManager to match its current responsibilities better. * The ELManager is no longer optional in the code (it won't have a nil value under any circumstances). * The support for subscribing for headers was removed as it only worked with WebSockets and contributed significant complexity while bringing only a very minor advantage. * The `--web3-url` parameter has been deprecated in favor of a new `--el` parameter. The new parameter has a reasonable default value and supports specifying a different JWT for each connection. Each connection can also be configured with a different set of responsibilities (e.g. download deposits, validate blocks and/or produce blocks). On the command-line, these properties can be configured through URL properties stored in the #anchor part of the URL. In TOML files, they come with a very natural syntax (althrough the URL scheme is also supported). * The previously scattered EL-related state and logic is now moved to `eth1_monitor.nim` (this module will be renamed to `el_manager.nim` in a follow-up commit). State is assigned properly either to the `ELManager` or the to individual `ELConnection` objects where appropriate. The ELManager executes all Engine API requests against all attached EL nodes, in parallel. It compares their results and if there is a disagreement regarding the validity of a certain payload, this is detected and the beacon node is protected from publishing a block with a potential execution layer consensus bug in it. The BN provides metrics per EL node for the number of successful or failed requests for each type Engine API requests. If an EL node goes offline and connectivity is resoted later, we report the problem and the remedy in edge-triggered fashion. * More progress towards implementing Deneb block production in the VC and comparing the value of blocks produced by the EL and the builder API. * Adds a Makefile target for the zhejiang testnet
2023-03-05 01:40:21 +00:00
let latestBlock = try:
More metrics for the EL monitor (#4707) * `engine_api_response_time` provides a histogram for the Engine API response times for each unique pair ot URL and request type. * All engine API requests are now tracked Other changes: The client will no longer exit on start-up if it fails to connect to a properly configured EL node.
2023-03-09 17:29:04 +00:00
connection.trackedRequestWithTimeout(
"getBlockByNumber",
Support for driving multiple EL nodes from a single Nimbus BN (#4465) * Support for driving multiple EL nodes from a single Nimbus BN Full list of changes: * Eth1Monitor has been renamed to ELManager to match its current responsibilities better. * The ELManager is no longer optional in the code (it won't have a nil value under any circumstances). * The support for subscribing for headers was removed as it only worked with WebSockets and contributed significant complexity while bringing only a very minor advantage. * The `--web3-url` parameter has been deprecated in favor of a new `--el` parameter. The new parameter has a reasonable default value and supports specifying a different JWT for each connection. Each connection can also be configured with a different set of responsibilities (e.g. download deposits, validate blocks and/or produce blocks). On the command-line, these properties can be configured through URL properties stored in the #anchor part of the URL. In TOML files, they come with a very natural syntax (althrough the URL scheme is also supported). * The previously scattered EL-related state and logic is now moved to `eth1_monitor.nim` (this module will be renamed to `el_manager.nim` in a follow-up commit). State is assigned properly either to the `ELManager` or the to individual `ELConnection` objects where appropriate. The ELManager executes all Engine API requests against all attached EL nodes, in parallel. It compares their results and if there is a disagreement regarding the validity of a certain payload, this is detected and the beacon node is protected from publishing a block with a potential execution layer consensus bug in it. The BN provides metrics per EL node for the number of successful or failed requests for each type Engine API requests. If an EL node goes offline and connectivity is resoted later, we report the problem and the remedy in edge-triggered fashion. * More progress towards implementing Deneb block production in the VC and comparing the value of blocks produced by the EL and the builder API. * Adds a Makefile target for the zhejiang testnet
2023-03-05 01:40:21 +00:00
rpcClient.eth_getBlockByNumber(blockId("latest"), false),
web3RequestsTimeout)
except CatchableError as err:
Cosmetic logging changes in the EL manager
2023-03-10 09:12:29 +00:00
warn "Failed to obtain the latest block from the EL", err = err.msg
Support for driving multiple EL nodes from a single Nimbus BN (#4465) * Support for driving multiple EL nodes from a single Nimbus BN Full list of changes: * Eth1Monitor has been renamed to ELManager to match its current responsibilities better. * The ELManager is no longer optional in the code (it won't have a nil value under any circumstances). * The support for subscribing for headers was removed as it only worked with WebSockets and contributed significant complexity while bringing only a very minor advantage. * The `--web3-url` parameter has been deprecated in favor of a new `--el` parameter. The new parameter has a reasonable default value and supports specifying a different JWT for each connection. Each connection can also be configured with a different set of responsibilities (e.g. download deposits, validate blocks and/or produce blocks). On the command-line, these properties can be configured through URL properties stored in the #anchor part of the URL. In TOML files, they come with a very natural syntax (althrough the URL scheme is also supported). * The previously scattered EL-related state and logic is now moved to `eth1_monitor.nim` (this module will be renamed to `el_manager.nim` in a follow-up commit). State is assigned properly either to the `ELManager` or the to individual `ELConnection` objects where appropriate. The ELManager executes all Engine API requests against all attached EL nodes, in parallel. It compares their results and if there is a disagreement regarding the validity of a certain payload, this is detected and the beacon node is protected from publishing a block with a potential execution layer consensus bug in it. The BN provides metrics per EL node for the number of successful or failed requests for each type Engine API requests. If an EL node goes offline and connectivity is resoted later, we report the problem and the remedy in edge-triggered fashion. * More progress towards implementing Deneb block production in the VC and comparing the value of blocks produced by the EL and the builder API. * Adds a Makefile target for the zhejiang testnet
2023-03-05 01:40:21 +00:00
raise err
Add polling support in the Eth1Monitor (extracted from the merge branch)
2021-11-25 16:51:51 +00:00
Support for driving multiple EL nodes from a single Nimbus BN (#4465) * Support for driving multiple EL nodes from a single Nimbus BN Full list of changes: * Eth1Monitor has been renamed to ELManager to match its current responsibilities better. * The ELManager is no longer optional in the code (it won't have a nil value under any circumstances). * The support for subscribing for headers was removed as it only worked with WebSockets and contributed significant complexity while bringing only a very minor advantage. * The `--web3-url` parameter has been deprecated in favor of a new `--el` parameter. The new parameter has a reasonable default value and supports specifying a different JWT for each connection. Each connection can also be configured with a different set of responsibilities (e.g. download deposits, validate blocks and/or produce blocks). On the command-line, these properties can be configured through URL properties stored in the #anchor part of the URL. In TOML files, they come with a very natural syntax (althrough the URL scheme is also supported). * The previously scattered EL-related state and logic is now moved to `eth1_monitor.nim` (this module will be renamed to `el_manager.nim` in a follow-up commit). State is assigned properly either to the `ELManager` or the to individual `ELConnection` objects where appropriate. The ELManager executes all Engine API requests against all attached EL nodes, in parallel. It compares their results and if there is a disagreement regarding the validity of a certain payload, this is detected and the beacon node is protected from publishing a block with a potential execution layer consensus bug in it. The BN provides metrics per EL node for the number of successful or failed requests for each type Engine API requests. If an EL node goes offline and connectivity is resoted later, we report the problem and the remedy in edge-triggered fashion. * More progress towards implementing Deneb block production in the VC and comparing the value of blocks produced by the EL and the builder API. * Adds a Makefile target for the zhejiang testnet
2023-03-05 01:40:21 +00:00
m.syncTargetBlock = some(
if Eth1BlockNumber(latestBlock.number) > m.cfg.ETH1_FOLLOW_DISTANCE:
Eth1BlockNumber(latestBlock.number) - m.cfg.ETH1_FOLLOW_DISTANCE
import EL deposits even when EL is stuck (#3956) * import EL deposits even when EL is stuck The `eth1_monitor` only starts importing deposits once the EL reports a new head block. However, the EL may be stuck at a block, e.g., the TTD. By polling the latest EL block once after subscribing to new EL block events it is ensured that deposits are still imported in this situation. * also poll once on re-connects * update `eth1_latest_head` metric in poll mode * add comment about similar polling vs events parts * replace check with assert * `isNewLastBlock` helper
2022-08-12 19:44:55 +00:00
else:
Support for driving multiple EL nodes from a single Nimbus BN (#4465) * Support for driving multiple EL nodes from a single Nimbus BN Full list of changes: * Eth1Monitor has been renamed to ELManager to match its current responsibilities better. * The ELManager is no longer optional in the code (it won't have a nil value under any circumstances). * The support for subscribing for headers was removed as it only worked with WebSockets and contributed significant complexity while bringing only a very minor advantage. * The `--web3-url` parameter has been deprecated in favor of a new `--el` parameter. The new parameter has a reasonable default value and supports specifying a different JWT for each connection. Each connection can also be configured with a different set of responsibilities (e.g. download deposits, validate blocks and/or produce blocks). On the command-line, these properties can be configured through URL properties stored in the #anchor part of the URL. In TOML files, they come with a very natural syntax (althrough the URL scheme is also supported). * The previously scattered EL-related state and logic is now moved to `eth1_monitor.nim` (this module will be renamed to `el_manager.nim` in a follow-up commit). State is assigned properly either to the `ELManager` or the to individual `ELConnection` objects where appropriate. The ELManager executes all Engine API requests against all attached EL nodes, in parallel. It compares their results and if there is a disagreement regarding the validity of a certain payload, this is detected and the beacon node is protected from publishing a block with a potential execution layer consensus bug in it. The BN provides metrics per EL node for the number of successful or failed requests for each type Engine API requests. If an EL node goes offline and connectivity is resoted later, we report the problem and the remedy in edge-triggered fashion. * More progress towards implementing Deneb block production in the VC and comparing the value of blocks produced by the EL and the builder API. * Adds a Makefile target for the zhejiang testnet
2023-03-05 01:40:21 +00:00
Eth1BlockNumber(0))
if m.syncTargetBlock.get <= eth1SyncedTo:
# The chain reorged to a lower height.
# It's relatively safe to ignore that.
await sleepAsync(m.cfg.SECONDS_PER_ETH1_BLOCK.int.seconds)
continue
Add polling support in the Eth1Monitor (extracted from the merge branch)
2021-11-25 16:51:51 +00:00
Support for driving multiple EL nodes from a single Nimbus BN (#4465) * Support for driving multiple EL nodes from a single Nimbus BN Full list of changes: * Eth1Monitor has been renamed to ELManager to match its current responsibilities better. * The ELManager is no longer optional in the code (it won't have a nil value under any circumstances). * The support for subscribing for headers was removed as it only worked with WebSockets and contributed significant complexity while bringing only a very minor advantage. * The `--web3-url` parameter has been deprecated in favor of a new `--el` parameter. The new parameter has a reasonable default value and supports specifying a different JWT for each connection. Each connection can also be configured with a different set of responsibilities (e.g. download deposits, validate blocks and/or produce blocks). On the command-line, these properties can be configured through URL properties stored in the #anchor part of the URL. In TOML files, they come with a very natural syntax (althrough the URL scheme is also supported). * The previously scattered EL-related state and logic is now moved to `eth1_monitor.nim` (this module will be renamed to `el_manager.nim` in a follow-up commit). State is assigned properly either to the `ELManager` or the to individual `ELConnection` objects where appropriate. The ELManager executes all Engine API requests against all attached EL nodes, in parallel. It compares their results and if there is a disagreement regarding the validity of a certain payload, this is detected and the beacon node is protected from publishing a block with a potential execution layer consensus bug in it. The BN provides metrics per EL node for the number of successful or failed requests for each type Engine API requests. If an EL node goes offline and connectivity is resoted later, we report the problem and the remedy in edge-triggered fashion. * More progress towards implementing Deneb block production in the VC and comparing the value of blocks produced by the EL and the builder API. * Adds a Makefile target for the zhejiang testnet
2023-03-05 01:40:21 +00:00
eth1_latest_head.set latestBlock.number.toGaugeValue
Tentative fix for the freezing Eth1 monitor activity
2021-02-04 15:01:47 +00:00
Support for driving multiple EL nodes from a single Nimbus BN (#4465) * Support for driving multiple EL nodes from a single Nimbus BN Full list of changes: * Eth1Monitor has been renamed to ELManager to match its current responsibilities better. * The ELManager is no longer optional in the code (it won't have a nil value under any circumstances). * The support for subscribing for headers was removed as it only worked with WebSockets and contributed significant complexity while bringing only a very minor advantage. * The `--web3-url` parameter has been deprecated in favor of a new `--el` parameter. The new parameter has a reasonable default value and supports specifying a different JWT for each connection. Each connection can also be configured with a different set of responsibilities (e.g. download deposits, validate blocks and/or produce blocks). On the command-line, these properties can be configured through URL properties stored in the #anchor part of the URL. In TOML files, they come with a very natural syntax (althrough the URL scheme is also supported). * The previously scattered EL-related state and logic is now moved to `eth1_monitor.nim` (this module will be renamed to `el_manager.nim` in a follow-up commit). State is assigned properly either to the `ELManager` or the to individual `ELConnection` objects where appropriate. The ELManager executes all Engine API requests against all attached EL nodes, in parallel. It compares their results and if there is a disagreement regarding the validity of a certain payload, this is detected and the beacon node is protected from publishing a block with a potential execution layer consensus bug in it. The BN provides metrics per EL node for the number of successful or failed requests for each type Engine API requests. If an EL node goes offline and connectivity is resoted later, we report the problem and the remedy in edge-triggered fashion. * More progress towards implementing Deneb block production in the VC and comparing the value of blocks produced by the EL and the builder API. * Adds a Makefile target for the zhejiang testnet
2023-03-05 01:40:21 +00:00
if shouldProcessDeposits and
latestBlock.number.uint64 > m.cfg.ETH1_FOLLOW_DISTANCE:
More metrics for the EL monitor (#4707) * `engine_api_response_time` provides a histogram for the Engine API response times for each unique pair ot URL and request type. * All engine API requests are now tracked Other changes: The client will no longer exit on start-up if it fails to connect to a properly configured EL node.
2023-03-09 17:29:04 +00:00
await m.syncBlockRange(connection,
rpcClient,
Support for driving multiple EL nodes from a single Nimbus BN (#4465) * Support for driving multiple EL nodes from a single Nimbus BN Full list of changes: * Eth1Monitor has been renamed to ELManager to match its current responsibilities better. * The ELManager is no longer optional in the code (it won't have a nil value under any circumstances). * The support for subscribing for headers was removed as it only worked with WebSockets and contributed significant complexity while bringing only a very minor advantage. * The `--web3-url` parameter has been deprecated in favor of a new `--el` parameter. The new parameter has a reasonable default value and supports specifying a different JWT for each connection. Each connection can also be configured with a different set of responsibilities (e.g. download deposits, validate blocks and/or produce blocks). On the command-line, these properties can be configured through URL properties stored in the #anchor part of the URL. In TOML files, they come with a very natural syntax (althrough the URL scheme is also supported). * The previously scattered EL-related state and logic is now moved to `eth1_monitor.nim` (this module will be renamed to `el_manager.nim` in a follow-up commit). State is assigned properly either to the `ELManager` or the to individual `ELConnection` objects where appropriate. The ELManager executes all Engine API requests against all attached EL nodes, in parallel. It compares their results and if there is a disagreement regarding the validity of a certain payload, this is detected and the beacon node is protected from publishing a block with a potential execution layer consensus bug in it. The BN provides metrics per EL node for the number of successful or failed requests for each type Engine API requests. If an EL node goes offline and connectivity is resoted later, we report the problem and the remedy in edge-triggered fashion. * More progress towards implementing Deneb block production in the VC and comparing the value of blocks produced by the EL and the builder API. * Adds a Makefile target for the zhejiang testnet
2023-03-05 01:40:21 +00:00
depositContract,
eth1SyncedTo + 1,
m.syncTargetBlock.get,
m.earliestBlockOfInterest(Eth1BlockNumber latestBlock.number))
Spec-compliant implementation of Eth1 monitoring; Eth1-enabled local sim BEWARE! This commit will trigger a stack overflow during local sim
2020-03-24 11:13:07 +00:00
Support for driving multiple EL nodes from a single Nimbus BN (#4465) * Support for driving multiple EL nodes from a single Nimbus BN Full list of changes: * Eth1Monitor has been renamed to ELManager to match its current responsibilities better. * The ELManager is no longer optional in the code (it won't have a nil value under any circumstances). * The support for subscribing for headers was removed as it only worked with WebSockets and contributed significant complexity while bringing only a very minor advantage. * The `--web3-url` parameter has been deprecated in favor of a new `--el` parameter. The new parameter has a reasonable default value and supports specifying a different JWT for each connection. Each connection can also be configured with a different set of responsibilities (e.g. download deposits, validate blocks and/or produce blocks). On the command-line, these properties can be configured through URL properties stored in the #anchor part of the URL. In TOML files, they come with a very natural syntax (althrough the URL scheme is also supported). * The previously scattered EL-related state and logic is now moved to `eth1_monitor.nim` (this module will be renamed to `el_manager.nim` in a follow-up commit). State is assigned properly either to the `ELManager` or the to individual `ELConnection` objects where appropriate. The ELManager executes all Engine API requests against all attached EL nodes, in parallel. It compares their results and if there is a disagreement regarding the validity of a certain payload, this is detected and the beacon node is protected from publishing a block with a potential execution layer consensus bug in it. The BN provides metrics per EL node for the number of successful or failed requests for each type Engine API requests. If an EL node goes offline and connectivity is resoted later, we report the problem and the remedy in edge-triggered fashion. * More progress towards implementing Deneb block production in the VC and comparing the value of blocks produced by the EL and the builder API. * Adds a Makefile target for the zhejiang testnet
2023-03-05 01:40:21 +00:00
eth1SyncedTo = m.syncTargetBlock.get
eth1_synced_head.set eth1SyncedTo.toGaugeValue
Bellatrix TTD detection (#3745) * Bellatrix TTD detection * Update beacon_chain/eth1/eth1_monitor.nim Co-authored-by: Etan Kissling <etan@status.im> * Update beacon_chain/nimbus_beacon_node.nim Co-authored-by: tersec <tersec@users.noreply.github.com> Co-authored-by: Etan Kissling <etan@status.im> Co-authored-by: tersec <tersec@users.noreply.github.com>
2022-06-15 02:38:27 +00:00
Support for driving multiple EL nodes from a single Nimbus BN (#4465) * Support for driving multiple EL nodes from a single Nimbus BN Full list of changes: * Eth1Monitor has been renamed to ELManager to match its current responsibilities better. * The ELManager is no longer optional in the code (it won't have a nil value under any circumstances). * The support for subscribing for headers was removed as it only worked with WebSockets and contributed significant complexity while bringing only a very minor advantage. * The `--web3-url` parameter has been deprecated in favor of a new `--el` parameter. The new parameter has a reasonable default value and supports specifying a different JWT for each connection. Each connection can also be configured with a different set of responsibilities (e.g. download deposits, validate blocks and/or produce blocks). On the command-line, these properties can be configured through URL properties stored in the #anchor part of the URL. In TOML files, they come with a very natural syntax (althrough the URL scheme is also supported). * The previously scattered EL-related state and logic is now moved to `eth1_monitor.nim` (this module will be renamed to `el_manager.nim` in a follow-up commit). State is assigned properly either to the `ELManager` or the to individual `ELConnection` objects where appropriate. The ELManager executes all Engine API requests against all attached EL nodes, in parallel. It compares their results and if there is a disagreement regarding the validity of a certain payload, this is detected and the beacon node is protected from publishing a block with a potential execution layer consensus bug in it. The BN provides metrics per EL node for the number of successful or failed requests for each type Engine API requests. If an EL node goes offline and connectivity is resoted later, we report the problem and the remedy in edge-triggered fashion. * More progress towards implementing Deneb block production in the VC and comparing the value of blocks produced by the EL and the builder API. * Adds a Makefile target for the zhejiang testnet
2023-03-05 01:40:21 +00:00
proc startChainSyncingLoop(m: ELManager) {.async.} =
simplify deposit syncing message (#5177) and make it less alarming since there's no action required from the user typically
2023-07-11 06:01:56 +00:00
info "Starting execution layer deposit syncing",
Support for driving multiple EL nodes from a single Nimbus BN (#4465) * Support for driving multiple EL nodes from a single Nimbus BN Full list of changes: * Eth1Monitor has been renamed to ELManager to match its current responsibilities better. * The ELManager is no longer optional in the code (it won't have a nil value under any circumstances). * The support for subscribing for headers was removed as it only worked with WebSockets and contributed significant complexity while bringing only a very minor advantage. * The `--web3-url` parameter has been deprecated in favor of a new `--el` parameter. The new parameter has a reasonable default value and supports specifying a different JWT for each connection. Each connection can also be configured with a different set of responsibilities (e.g. download deposits, validate blocks and/or produce blocks). On the command-line, these properties can be configured through URL properties stored in the #anchor part of the URL. In TOML files, they come with a very natural syntax (althrough the URL scheme is also supported). * The previously scattered EL-related state and logic is now moved to `eth1_monitor.nim` (this module will be renamed to `el_manager.nim` in a follow-up commit). State is assigned properly either to the `ELManager` or the to individual `ELConnection` objects where appropriate. The ELManager executes all Engine API requests against all attached EL nodes, in parallel. It compares their results and if there is a disagreement regarding the validity of a certain payload, this is detected and the beacon node is protected from publishing a block with a potential execution layer consensus bug in it. The BN provides metrics per EL node for the number of successful or failed requests for each type Engine API requests. If an EL node goes offline and connectivity is resoted later, we report the problem and the remedy in edge-triggered fashion. * More progress towards implementing Deneb block production in the VC and comparing the value of blocks produced by the EL and the builder API. * Adds a Makefile target for the zhejiang testnet
2023-03-05 01:40:21 +00:00
contract = $m.depositContractAddress
Spec-compliant implementation of Eth1 monitoring; Eth1-enabled local sim BEWARE! This commit will trigger a stack overflow during local sim
2020-03-24 11:13:07 +00:00
Prevent the build up of parallel 'waitELToSync' operations
2023-03-10 10:55:55 +00:00
var syncedConnectionFut = m.selectConnectionForChainSyncing()
info "Connection attempt started"
Support for driving multiple EL nodes from a single Nimbus BN (#4465) * Support for driving multiple EL nodes from a single Nimbus BN Full list of changes: * Eth1Monitor has been renamed to ELManager to match its current responsibilities better. * The ELManager is no longer optional in the code (it won't have a nil value under any circumstances). * The support for subscribing for headers was removed as it only worked with WebSockets and contributed significant complexity while bringing only a very minor advantage. * The `--web3-url` parameter has been deprecated in favor of a new `--el` parameter. The new parameter has a reasonable default value and supports specifying a different JWT for each connection. Each connection can also be configured with a different set of responsibilities (e.g. download deposits, validate blocks and/or produce blocks). On the command-line, these properties can be configured through URL properties stored in the #anchor part of the URL. In TOML files, they come with a very natural syntax (althrough the URL scheme is also supported). * The previously scattered EL-related state and logic is now moved to `eth1_monitor.nim` (this module will be renamed to `el_manager.nim` in a follow-up commit). State is assigned properly either to the `ELManager` or the to individual `ELConnection` objects where appropriate. The ELManager executes all Engine API requests against all attached EL nodes, in parallel. It compares their results and if there is a disagreement regarding the validity of a certain payload, this is detected and the beacon node is protected from publishing a block with a potential execution layer consensus bug in it. The BN provides metrics per EL node for the number of successful or failed requests for each type Engine API requests. If an EL node goes offline and connectivity is resoted later, we report the problem and the remedy in edge-triggered fashion. * More progress towards implementing Deneb block production in the VC and comparing the value of blocks produced by the EL and the builder API. * Adds a Makefile target for the zhejiang testnet
2023-03-05 01:40:21 +00:00
while true:
Prevent the build up of parallel 'waitELToSync' operations
2023-03-10 10:55:55 +00:00
try:
await syncedConnectionFut or sleepAsync(60.seconds)
if not syncedConnectionFut.finished:
simplify deposit syncing message (#5177) and make it less alarming since there's no action required from the user typically
2023-07-11 06:01:56 +00:00
notice "No synced execution layer available for deposit syncing"
Support for driving multiple EL nodes from a single Nimbus BN (#4465) * Support for driving multiple EL nodes from a single Nimbus BN Full list of changes: * Eth1Monitor has been renamed to ELManager to match its current responsibilities better. * The ELManager is no longer optional in the code (it won't have a nil value under any circumstances). * The support for subscribing for headers was removed as it only worked with WebSockets and contributed significant complexity while bringing only a very minor advantage. * The `--web3-url` parameter has been deprecated in favor of a new `--el` parameter. The new parameter has a reasonable default value and supports specifying a different JWT for each connection. Each connection can also be configured with a different set of responsibilities (e.g. download deposits, validate blocks and/or produce blocks). On the command-line, these properties can be configured through URL properties stored in the #anchor part of the URL. In TOML files, they come with a very natural syntax (althrough the URL scheme is also supported). * The previously scattered EL-related state and logic is now moved to `eth1_monitor.nim` (this module will be renamed to `el_manager.nim` in a follow-up commit). State is assigned properly either to the `ELManager` or the to individual `ELConnection` objects where appropriate. The ELManager executes all Engine API requests against all attached EL nodes, in parallel. It compares their results and if there is a disagreement regarding the validity of a certain payload, this is detected and the beacon node is protected from publishing a block with a potential execution layer consensus bug in it. The BN provides metrics per EL node for the number of successful or failed requests for each type Engine API requests. If an EL node goes offline and connectivity is resoted later, we report the problem and the remedy in edge-triggered fashion. * More progress towards implementing Deneb block production in the VC and comparing the value of blocks produced by the EL and the builder API. * Adds a Makefile target for the zhejiang testnet
2023-03-05 01:40:21 +00:00
await sleepAsync(chronos.seconds(30))
Bellatrix TTD detection (#3745) * Bellatrix TTD detection * Update beacon_chain/eth1/eth1_monitor.nim Co-authored-by: Etan Kissling <etan@status.im> * Update beacon_chain/nimbus_beacon_node.nim Co-authored-by: tersec <tersec@users.noreply.github.com> Co-authored-by: Etan Kissling <etan@status.im> Co-authored-by: tersec <tersec@users.noreply.github.com>
2022-06-15 02:38:27 +00:00
continue
Prevent the build up of parallel 'waitELToSync' operations
2023-03-10 10:55:55 +00:00
await syncEth1Chain(m, syncedConnectionFut.read)
Support for driving multiple EL nodes from a single Nimbus BN (#4465) * Support for driving multiple EL nodes from a single Nimbus BN Full list of changes: * Eth1Monitor has been renamed to ELManager to match its current responsibilities better. * The ELManager is no longer optional in the code (it won't have a nil value under any circumstances). * The support for subscribing for headers was removed as it only worked with WebSockets and contributed significant complexity while bringing only a very minor advantage. * The `--web3-url` parameter has been deprecated in favor of a new `--el` parameter. The new parameter has a reasonable default value and supports specifying a different JWT for each connection. Each connection can also be configured with a different set of responsibilities (e.g. download deposits, validate blocks and/or produce blocks). On the command-line, these properties can be configured through URL properties stored in the #anchor part of the URL. In TOML files, they come with a very natural syntax (althrough the URL scheme is also supported). * The previously scattered EL-related state and logic is now moved to `eth1_monitor.nim` (this module will be renamed to `el_manager.nim` in a follow-up commit). State is assigned properly either to the `ELManager` or the to individual `ELConnection` objects where appropriate. The ELManager executes all Engine API requests against all attached EL nodes, in parallel. It compares their results and if there is a disagreement regarding the validity of a certain payload, this is detected and the beacon node is protected from publishing a block with a potential execution layer consensus bug in it. The BN provides metrics per EL node for the number of successful or failed requests for each type Engine API requests. If an EL node goes offline and connectivity is resoted later, we report the problem and the remedy in edge-triggered fashion. * More progress towards implementing Deneb block production in the VC and comparing the value of blocks produced by the EL and the builder API. * Adds a Makefile target for the zhejiang testnet
2023-03-05 01:40:21 +00:00
except CatchableError as err:
Prevent the build up of parallel 'waitELToSync' operations
2023-03-10 10:55:55 +00:00
await sleepAsync(10.seconds)
More forceful EL connection resetting (fixes #4424) (#4708)
2023-03-09 23:41:28 +00:00
# A more detailed error is already logged by trackEngineApiRequest
debug "Restarting the deposit syncing loop"
Prevent the build up of parallel 'waitELToSync' operations
2023-03-10 10:55:55 +00:00
# To be extra safe, we will make a fresh connection attempt
await syncedConnectionFut.cancelAndWait()
syncedConnectionFut = m.selectConnectionForChainSyncing()
Support for driving multiple EL nodes from a single Nimbus BN (#4465) * Support for driving multiple EL nodes from a single Nimbus BN Full list of changes: * Eth1Monitor has been renamed to ELManager to match its current responsibilities better. * The ELManager is no longer optional in the code (it won't have a nil value under any circumstances). * The support for subscribing for headers was removed as it only worked with WebSockets and contributed significant complexity while bringing only a very minor advantage. * The `--web3-url` parameter has been deprecated in favor of a new `--el` parameter. The new parameter has a reasonable default value and supports specifying a different JWT for each connection. Each connection can also be configured with a different set of responsibilities (e.g. download deposits, validate blocks and/or produce blocks). On the command-line, these properties can be configured through URL properties stored in the #anchor part of the URL. In TOML files, they come with a very natural syntax (althrough the URL scheme is also supported). * The previously scattered EL-related state and logic is now moved to `eth1_monitor.nim` (this module will be renamed to `el_manager.nim` in a follow-up commit). State is assigned properly either to the `ELManager` or the to individual `ELConnection` objects where appropriate. The ELManager executes all Engine API requests against all attached EL nodes, in parallel. It compares their results and if there is a disagreement regarding the validity of a certain payload, this is detected and the beacon node is protected from publishing a block with a potential execution layer consensus bug in it. The BN provides metrics per EL node for the number of successful or failed requests for each type Engine API requests. If an EL node goes offline and connectivity is resoted later, we report the problem and the remedy in edge-triggered fashion. * More progress towards implementing Deneb block production in the VC and comparing the value of blocks produced by the EL and the builder API. * Adds a Makefile target for the zhejiang testnet
2023-03-05 01:40:21 +00:00
ensure LC runs `exchangeTransitionConfiguration` loop (#5312) In #4465, a regression was introduced by deleting the periodic call to `engine_exchangeTransitionConfiguration` in the Nimbus light client. This led to the light client never treating the EL as online and, subsequently, not sending `engine_newPayload` requests for new blocks. Strangely, `engine_forkchoiceUpdated` requests still make it through :) Geth still requires both `engine_newPayload` and `fcU` to be called. By restoring the `exchangeTransitionConfiguration` loop, `newPayload` requests are once more issued to the EL.
2023-08-18 09:30:02 +00:00
proc start*(m: ELManager, syncChain = true) {.gcsafe.} =
Support for driving multiple EL nodes from a single Nimbus BN (#4465) * Support for driving multiple EL nodes from a single Nimbus BN Full list of changes: * Eth1Monitor has been renamed to ELManager to match its current responsibilities better. * The ELManager is no longer optional in the code (it won't have a nil value under any circumstances). * The support for subscribing for headers was removed as it only worked with WebSockets and contributed significant complexity while bringing only a very minor advantage. * The `--web3-url` parameter has been deprecated in favor of a new `--el` parameter. The new parameter has a reasonable default value and supports specifying a different JWT for each connection. Each connection can also be configured with a different set of responsibilities (e.g. download deposits, validate blocks and/or produce blocks). On the command-line, these properties can be configured through URL properties stored in the #anchor part of the URL. In TOML files, they come with a very natural syntax (althrough the URL scheme is also supported). * The previously scattered EL-related state and logic is now moved to `eth1_monitor.nim` (this module will be renamed to `el_manager.nim` in a follow-up commit). State is assigned properly either to the `ELManager` or the to individual `ELConnection` objects where appropriate. The ELManager executes all Engine API requests against all attached EL nodes, in parallel. It compares their results and if there is a disagreement regarding the validity of a certain payload, this is detected and the beacon node is protected from publishing a block with a potential execution layer consensus bug in it. The BN provides metrics per EL node for the number of successful or failed requests for each type Engine API requests. If an EL node goes offline and connectivity is resoted later, we report the problem and the remedy in edge-triggered fashion. * More progress towards implementing Deneb block production in the VC and comparing the value of blocks produced by the EL and the builder API. * Adds a Makefile target for the zhejiang testnet
2023-03-05 01:40:21 +00:00
if m.elConnections.len == 0:
return
## Calling `ELManager.start()` on an already started ELManager is a noop
ensure LC runs `exchangeTransitionConfiguration` loop (#5312) In #4465, a regression was introduced by deleting the periodic call to `engine_exchangeTransitionConfiguration` in the Nimbus light client. This led to the light client never treating the EL as online and, subsequently, not sending `engine_newPayload` requests for new blocks. Strangely, `engine_forkchoiceUpdated` requests still make it through :) Geth still requires both `engine_newPayload` and `fcU` to be called. By restoring the `exchangeTransitionConfiguration` loop, `newPayload` requests are once more issued to the EL.
2023-08-18 09:30:02 +00:00
if syncChain and m.chainSyncingLoopFut.isNil:
Support for driving multiple EL nodes from a single Nimbus BN (#4465) * Support for driving multiple EL nodes from a single Nimbus BN Full list of changes: * Eth1Monitor has been renamed to ELManager to match its current responsibilities better. * The ELManager is no longer optional in the code (it won't have a nil value under any circumstances). * The support for subscribing for headers was removed as it only worked with WebSockets and contributed significant complexity while bringing only a very minor advantage. * The `--web3-url` parameter has been deprecated in favor of a new `--el` parameter. The new parameter has a reasonable default value and supports specifying a different JWT for each connection. Each connection can also be configured with a different set of responsibilities (e.g. download deposits, validate blocks and/or produce blocks). On the command-line, these properties can be configured through URL properties stored in the #anchor part of the URL. In TOML files, they come with a very natural syntax (althrough the URL scheme is also supported). * The previously scattered EL-related state and logic is now moved to `eth1_monitor.nim` (this module will be renamed to `el_manager.nim` in a follow-up commit). State is assigned properly either to the `ELManager` or the to individual `ELConnection` objects where appropriate. The ELManager executes all Engine API requests against all attached EL nodes, in parallel. It compares their results and if there is a disagreement regarding the validity of a certain payload, this is detected and the beacon node is protected from publishing a block with a potential execution layer consensus bug in it. The BN provides metrics per EL node for the number of successful or failed requests for each type Engine API requests. If an EL node goes offline and connectivity is resoted later, we report the problem and the remedy in edge-triggered fashion. * More progress towards implementing Deneb block production in the VC and comparing the value of blocks produced by the EL and the builder API. * Adds a Makefile target for the zhejiang testnet
2023-03-05 01:40:21 +00:00
m.chainSyncingLoopFut =
m.startChainSyncingLoop()
if m.hasJwtSecret and m.exchangeTransitionConfigurationLoopFut.isNil:
m.exchangeTransitionConfigurationLoopFut =
m.startExchangeTransitionConfigurationLoop()
Better connection error handling in MainchainMonitor
2019-11-22 13:16:07 +00:00
Avoid potential stalls in the Eth1 monitor code due to missing timeouts (#3905) * Avoid potential stalls in the Eth1 monitor code due to missing timeouts * 1 second timeout for exchange-configuration
2022-07-25 19:23:02 +00:00
func `$`(x: Quantity): string =
$(x.uint64)
func `$`(x: BlockObject): string =
$(x.number) & " [" & $(x.hash) & "]"
Add a 'we3 test' command for verifying the compatibility of a web3 provider
2020-12-04 16:28:42 +00:00
proc testWeb3Provider*(web3Url: Uri,
JWT support (#3561)
2022-03-31 14:43:05 +00:00
depositContractAddress: Eth1Address,
jwtSecret: Option[seq[byte]]) {.async.} =
Avoid potential stalls in the Eth1 monitor code due to missing timeouts (#3905) * Avoid potential stalls in the Eth1 monitor code due to missing timeouts * 1 second timeout for exchange-configuration
2022-07-25 19:23:02 +00:00
stdout.write "Establishing web3 connection..."
var web3: Web3
try:
web3 = awaitOrRaiseOnTimeout(
More forceful EL connection resetting (fixes #4424) (#4708)
2023-03-09 23:41:28 +00:00
newWeb3($web3Url, getJsonRpcRequestHeaders(jwtSecret)),
5.seconds)
Avoid potential stalls in the Eth1 monitor code due to missing timeouts (#3905) * Avoid potential stalls in the Eth1 monitor code due to missing timeouts * 1 second timeout for exchange-configuration
2022-07-25 19:23:02 +00:00
stdout.write "\rEstablishing web3 connection: Connected\n"
except CatchableError as err:
stdout.write "\rEstablishing web3 connection: Failure(" & err.msg & ")\n"
quit 1
template request(actionDesc: static string,
action: untyped): untyped =
stdout.write actionDesc & "..."
stdout.flushFile()
var res: typeof(read action)
try:
Support for driving multiple EL nodes from a single Nimbus BN (#4465) * Support for driving multiple EL nodes from a single Nimbus BN Full list of changes: * Eth1Monitor has been renamed to ELManager to match its current responsibilities better. * The ELManager is no longer optional in the code (it won't have a nil value under any circumstances). * The support for subscribing for headers was removed as it only worked with WebSockets and contributed significant complexity while bringing only a very minor advantage. * The `--web3-url` parameter has been deprecated in favor of a new `--el` parameter. The new parameter has a reasonable default value and supports specifying a different JWT for each connection. Each connection can also be configured with a different set of responsibilities (e.g. download deposits, validate blocks and/or produce blocks). On the command-line, these properties can be configured through URL properties stored in the #anchor part of the URL. In TOML files, they come with a very natural syntax (althrough the URL scheme is also supported). * The previously scattered EL-related state and logic is now moved to `eth1_monitor.nim` (this module will be renamed to `el_manager.nim` in a follow-up commit). State is assigned properly either to the `ELManager` or the to individual `ELConnection` objects where appropriate. The ELManager executes all Engine API requests against all attached EL nodes, in parallel. It compares their results and if there is a disagreement regarding the validity of a certain payload, this is detected and the beacon node is protected from publishing a block with a potential execution layer consensus bug in it. The BN provides metrics per EL node for the number of successful or failed requests for each type Engine API requests. If an EL node goes offline and connectivity is resoted later, we report the problem and the remedy in edge-triggered fashion. * More progress towards implementing Deneb block production in the VC and comparing the value of blocks produced by the EL and the builder API. * Adds a Makefile target for the zhejiang testnet
2023-03-05 01:40:21 +00:00
res = awaitOrRaiseOnTimeout(action, web3RequestsTimeout)
Avoid potential stalls in the Eth1 monitor code due to missing timeouts (#3905) * Avoid potential stalls in the Eth1 monitor code due to missing timeouts * 1 second timeout for exchange-configuration
2022-07-25 19:23:02 +00:00
stdout.write "\r" & actionDesc & ": " & $res
Add a 'we3 test' command for verifying the compatibility of a web3 provider
2020-12-04 16:28:42 +00:00
except CatchableError as err:
Avoid potential stalls in the Eth1 monitor code due to missing timeouts (#3905) * Avoid potential stalls in the Eth1 monitor code due to missing timeouts * 1 second timeout for exchange-configuration
2022-07-25 19:23:02 +00:00
stdout.write "\r" & actionDesc & ": Error(" & err.msg & ")"
stdout.write "\n"
res
Add a 'we3 test' command for verifying the compatibility of a web3 provider
2020-12-04 16:28:42 +00:00
let
Avoid potential stalls in the Eth1 monitor code due to missing timeouts (#3905) * Avoid potential stalls in the Eth1 monitor code due to missing timeouts * 1 second timeout for exchange-configuration
2022-07-25 19:23:02 +00:00
chainId = request "Chain ID":
web3.provider.eth_chainId()
use eth_chainId instead of net_version (#3804) * use eth_chainId instead of net_version * Update beacon_chain/eth1/eth1_monitor.nim Co-authored-by: Etan Kissling <etan@status.im> * fix logging for Quantity types Co-authored-by: Etan Kissling <etan@status.im>
2022-06-27 15:02:12 +00:00
Avoid potential stalls in the Eth1 monitor code due to missing timeouts (#3905) * Avoid potential stalls in the Eth1 monitor code due to missing timeouts * 1 second timeout for exchange-configuration
2022-07-25 19:23:02 +00:00
latestBlock = request "Latest block":
web3.provider.eth_getBlockByNumber(blockId("latest"), false)
syncStatus = request "Sync status":
web3.provider.eth_syncing()
ns = web3.contractSender(DepositContract, depositContractAddress)
depositRoot = request "Deposit root":
ns.get_deposit_root.call(blockNumber = latestBlock.number.uint64)