Commit Graph

550 Commits

Author SHA1 Message Date
Etan Kissling 2dbe24c740
move split view catchup to research branch (#6133)
Using a dedicated branch for researching the effectiveness of split view
scenario handling simplifies testing and avoids having partial work on
`unstable`. If we want, we can reintroduce it under a `--debug` flag at
a later time. But for now, Goerli is a rare opoprtunity to test this,
maybe just for another week or so.

- https://github.com/status-im/infra-nimbus/pull/179
2024-03-25 19:09:31 +01:00
Etan Kissling fc9bc1da3a
add branch discovery module for supporting chain stall situation (#6125)
In split view situation, the canonical chain may only be served by a
tiny amount of peers, and branches may span long durations. Minority
branches may still have a large weight from attestations and should
be discovered. To assist with that, add a branch discovery module that
assists in such a situation by specifically targeting peers with unknown
histories and downloading from them, in addition to sync manager work
which handles popular branches.
2024-03-24 08:41:47 +00:00
Etan Kissling 5d42859176
make `Gwei` `distinct` (#6090)
#6087 introduced a subtle change to `nim-web3` resulting in `Gwei` to be
serialized differently than before. Using a `distinct` type for `Gwei`
improves type safety and avoids such problems in the future.
2024-03-19 14:22:07 +01:00
Etan Kissling 8bd8ffe2bb
align default `syncHorizon` computation logic across networks (#6066)
The `syncHorizon` describes the number of empty slots before the beacon
node considers itself to be out of sync. There are two places where we
currently set this to 50 slots, but it makes more sense to base it on
wall time, e.g., the 10 minutes that the default 50 are derived from.
2024-03-12 21:51:18 +01:00
tersec 69af8f943e
implement blob_sidecar Beacon API streaming (#5728) 2024-01-13 11:52:13 +02:00
tersec 251143fd51
attest to known valid block when possible (#5313)
* attest to known valid block when possible

* cleaner approach; slot is always == attestation slot itself

* copyright year linting
2024-01-11 22:34:10 +00:00
Jacek Sieka 62cbdeefc5
verify `genesis_time` more strictly (fixes #1667) (#5694)
Bogus values lead to crashes down the line when timers overflow
2024-01-06 15:26:56 +01:00
Etan Kissling 030226148d
rename `exit_pool` > `validator_change_pool` (#5679)
The `ExitPool` was renamed to `ValidatorChangePool` with Capella, but
the files were still using the previous name. Rename for consistency.
2023-12-23 06:55:47 +01:00
Etan Kissling 583782a061
emit `proposer_slashing`/`attester_slashing` SSE on beacon-API (#5678)
Add support for slashings on the beacon-API event stream for compat with
beacon-API specs.

- https://github.com/ethereum/beacon-APIs/pull/376
2023-12-22 18:54:55 +01:00
Etan Kissling a2081521f6
emit `bls_to_execution_change` SSE on beacon-API (#5677)
With Capella, `bls_to_execution_change` SSE should be emitted on the
event stream whenever a new `SignedBLSToExecutionChange` is received.
Add this missing functionality for compatibility with beacon-API specs.

- https://github.com/ethereum/beacon-APIs/pull/248
2023-12-22 14:52:43 +01:00
Jacek Sieka 9d155ebb35
avoid json-rpc server import (#5649) 2023-12-05 22:08:18 +01:00
tersec 351472a2d4
per payload-builder payload builder validator registration (#5431) 2023-09-20 17:00:37 +00:00
Etan Kissling 6a2bac5cee
need even more log lines for debugging keymanager (#5260)
Keymanager test logs are still cut off, further increase log lines.
2023-08-07 18:02:08 +02:00
Eugene Kabanov c2c5d80a4f
VC: Remote BN clock offset monitoring. (#4846)
* Initial commit.

* Add algorithm in comment.
Remove delays.
Fix logging statement issues.
Change update from epoch to slot.

* Obtain timestamp earlier.

* Add processing delays into algorithm.

* Fix time offset logging to produce integers instead of strings.

* Address review comments.

* Fix copyright year.
Fix updateStatus().

* Remove fields from Slot start log statement.
Fix issues when BN do not support Nimbus Extensions.
Rename metric name and type change.

* Add beacon role to disable time offset check manually.
2023-06-28 15:33:07 +02:00
tersec ba597ef0a2
per-validator payload builder configuration (#5062) 2023-06-25 12:00:17 +00:00
tersec e8e67ec771
allow payload builder client to be function of validator/proposer (#5015)
* allow payload builder client to be function of validator/proposer

* fileExists has side effects on Windows and only Windows

* another not-always-func
2023-06-02 11:06:33 +00:00
tersec d1941b670a
refactor payload builder REST client usage (#4973)
* refactor payload builder REST client usage

* change HTTP response code
2023-05-25 18:38:56 +03:00
henridf 573228ffa0
Rename eth1/ -> el/ and eth1_monitor.nim -> el_monitor.nim (#4944) 2023-05-15 05:05:12 +00:00
henridf ef0b95dfbc
RequestManager: add support for fetching Blobs (#4844)
* RequestManager: add support for fetching Blobs

* Review feedback

* Lint

* Change peekSortedBlobless -> peekBlobless
2023-04-28 12:57:35 +00:00
zah 8771e91d53
Support for driving multiple EL nodes from a single Nimbus BN (#4465)
* Support for driving multiple EL nodes from a single Nimbus BN

Full list of changes:

* Eth1Monitor has been renamed to ELManager to match its current
  responsibilities better.

* The ELManager is no longer optional in the code (it won't have
  a nil value under any circumstances).

* The support for subscribing for headers was removed as it only
  worked with WebSockets and contributed significant complexity
  while bringing only a very minor advantage.

* The `--web3-url` parameter has been deprecated in favor of a
  new `--el` parameter. The new parameter has a reasonable default
  value and supports specifying a different JWT for each connection.
  Each connection can also be configured with a different set of
  responsibilities (e.g. download deposits, validate blocks and/or
  produce blocks). On the command-line, these properties can be
  configured through URL properties stored in the #anchor part of
  the URL. In TOML files, they come with a very natural syntax
  (althrough the URL scheme is also supported).

* The previously scattered EL-related state and logic is now moved
  to `eth1_monitor.nim` (this module will be renamed to `el_manager.nim`
  in a follow-up commit). State is assigned properly either to the
  `ELManager` or the to individual `ELConnection` objects where
  appropriate.

  The ELManager executes all Engine API requests against all attached
  EL nodes, in parallel. It compares their results and if there is a
  disagreement regarding the validity of a certain payload, this is
  detected and the beacon node is protected from publishing a block
  with a potential execution layer consensus bug in it.

  The BN provides metrics per EL node for the number of successful or
  failed requests for each type Engine API requests. If an EL node
  goes offline and connectivity is resoted later, we report the
  problem and the remedy in edge-triggered fashion.

* More progress towards implementing Deneb block production in the VC
  and comparing the value of blocks produced by the EL and the builder
  API.

* Adds a Makefile target for the zhejiang testnet
2023-03-05 01:40:21 +00:00
zah 6036f2e7d7
Local sim impovements (#4551)
* Local sim impovements

* Added support for running Capella and EIP-4844 simulations
  by downloading the correct version of Geth.

* Added support for using Nimbus remote signer and Web3Signer.
  Use 2 out of 3 threshold signing configuration in the mainnet
  configuration and regular remote signing in the minimal one.

* The local testnet simulation can now use a payload builder.
  This is currently not activated in CI due to lack of automated
  procedures for installing third-party relays or builders.

  You are adviced to use mergemock for now, but for most realistic
  results, we can create a simple builder based on the nimbus-eth1
  codebase that will be able to propose transactions from the regular
  network mempool.

* Start the simulation from a merged state. This would allow us
  to start removing pre-merge functionality such as the gossip
  subsciption logic. The commit also removes the merge-forcing
  hack installed after the TTD removal.

* Consolidate all the tools used in the local simulation into a
  single `ncli_testnet` binary.
2023-02-23 02:10:07 +00:00
Eugene Kabanov e91415662b
Keystore cache implementation. (#4372) 2023-02-16 19:25:48 +02:00
Jacek Sieka 856fcea8d7
fix slow checking of unknown validators (#4590)
We do a linear scan of all pubkeys for each validator and slot - this
becomes expensive with large validator counts.

* normalise BN/VC validator startup logging
* fix crash when host cannot be resolved while adding remote validator
* silence repeated log spam for unknown validators
* print pubkey/index/activation mapping on startup/validator
identification
2023-02-07 14:53:36 +00:00
tersec 819e007689
exit/validatorchange pool includes BLS to execution messages; REST support for new pool (#4519)
* exit/validatorchange pool includes BLS to execution messages; REST
support for new pool

* catch failed individual futures

* increase BLS changes bound and keep BLS seen consistent with subpool

* deque capacities should be powers of 2
2023-01-19 22:00:40 +00:00
tersec aea7a0c8b8
remove TTD monitoring (#4486) 2023-01-18 16:01:49 +02:00
Etan Kissling 7e276937dc
make LC data fork aware (#4493)
In a future fork, light client data will be extended with execution info
to support more use cases. To anticipate such an upgrade, introduce
`Forky` and `Forked` types, and ready the database schema.
Because the mapping of sync committee periods to fork versions is not
necessarily unique (fork schedule not in sync with period boundaries),
an additional column is added to `period` -> `LightClientUpdate` table.
2023-01-12 18:11:38 +01:00
tersec 5b0b48f6e9
implement /eth/v1/validator/register_validator (#4115) 2022-09-13 14:52:26 +03:00
tersec bf3a014287
more efficient forkchoiceUpdated usage (#4055)
* more efficient forkchoiceUpdated usage

* await rather than asyncSpawn; ensure head update before dag.updateHead

* use action tracker rather than attached validators to check for next slot proposal; use wall slot + 1 rather than state slot + 1 to correctly check when missing blocks

* re-add two-fcU case for when newPayload not VALID

* check dynamicFeeRecipientsStore for potential proposal

* remove duplicate checks for whether next proposer
2022-09-07 20:34:52 +02:00
Etan Kissling 9180f09641
reduce LC optsync latency (#4002)
The optimistic sync spec was updated since the LC based optsync module
was introduced. It is no longer necessary to wait for the justified
checkpoint to have execution enabled; instead, any block is okay to be
optimistically imported to the EL client, as long as its parent block
has execution enabled. Complex syncing logic has been removed, and the
LC optsync module will now follow gossip directly, reducing the latency
when using this module. Note that because this is now based on gossip
instead of using sync manager / request manager, that individual blocks
may be missed. However, EL clients should recover from this by fetching
missing blocks themselves.
2022-08-25 03:53:59 +00:00
tersec 1d55743ebb
allow execution clients several seconds to construct blocks (#4012) 2022-08-23 19:19:52 +03:00
zah fca20e08d6
Keymanager API for the validator client (#3976)
* Keymanager API for the validator client
* Properly treat the 'description' field as optional when loading Keystores
* Spec-compliant serialization of the slashing data in Keymanager's DeleteKeys response ()

Fixes #3940
Fixes #3964
Closes #3884 by adding test
2022-08-19 13:30:07 +03:00
tersec d62d13a23c
MEV block proposal (#3883)
* MEV validator registration

* add nearby canary to detect new beacon chain forks

* remove special MEV graffiti

* web3signer support

* fix trace logging

* Nim 1.2 needs raises Defect

* use template rather than proc in REST JSON parsing

* use --payload-builder-enable and --payload-builder-url

* explicitly default MEV to disabled

* explicitly empty default value for payload builder URL

* revert attestation pool to unstable version
2022-08-01 09:41:47 +03:00
zah cd04f27c37
Implement the /eth/v1/validator/prepare_beacon_proposer end-point (#3901) 2022-07-25 23:12:53 +03:00
Etan Kissling d6adf09e65
cleanup magic strings (#3891)
Replaces hardcoded IP address strings with constants.
Adjusts some LC log levels.
Removes no longer used `RpcServer` type.
2022-07-20 18:17:21 +00:00
tersec 06c8e10ae2
move consensus_manager to consensus_object_pools (#3852) 2022-07-13 14:13:54 +00:00
Jacek Sieka e1830519a4
Introduce message router (#3829)
Whether new blocks/attestations/etc are produced internally or received
via REST, their journey through the node is the same - to ensure that
they get the same treatment (logging, metrics, processing), this PR
moves the routing to a dedicated module and fixes several small
differences that existed before.

* `xxxValidator` -> `processMessageName` - the processor also was adding
messages to pools, so we want the name to reflect that action
* add missing "sent" metrics for some messages
* document ignore policy better - already-seen messages are not actaully
rebroadcast by libp2p
* skip redundant signature checks for internal validators consistently
2022-07-06 16:11:44 +00:00
Eugene Kabanov eb6b7affee
Add the `execution_optimistic` flag to REST API responses. (#3780)
* Initial commit

* Make `events` API spec compliant.

* Add `Eth-Consensus-Version` in responses.

* Bump chronos to get redirect with headers working.

* Add `is_optimistic` field and handling to syncing RestSyncInfo.
2022-06-20 08:53:39 +03:00
Etan Kissling 61ee0611bb
adopt LC REST API with v0 suffix (without proofs) (#3775)
* adopt LC REST API with v0 suffix (without proofs)

Adopts the light client data REST API used by Lodestar as defined in
https://github.com/ethereum/beacon-APIs/pull/181 with a v0 suffix.

Requests:
- `/eth/v0/beacon/light_client/bootstrap/{block_root}`
- `/eth/v0/beacon/light_client/updates?start_period={start_period}&count={count}`
- `/eth/v0/beacon/light_client/finality_update`
- `/eth/v0/beacon/light_client/optimistic_update`

HTTP Server-Sent Events (SSE):
- `light_client_finality_update_v0`
- `light_client_optimistic_update_v0`

More work is needed to adopt the proofs endpoint, it is not included.

* initialize event queues

* register event topics
2022-06-19 08:57:52 +03:00
Eugene Kabanov 5e7b28cd66
AsyncEventBus missing attestations fix. (#3664) 2022-06-17 18:27:28 +03:00
zah 694b653757
Bellatrix TTD detection (#3745)
* Bellatrix TTD detection

* Update beacon_chain/eth1/eth1_monitor.nim

Co-authored-by: Etan Kissling <etan@status.im>

* Update beacon_chain/nimbus_beacon_node.nim

Co-authored-by: tersec <tersec@users.noreply.github.com>

Co-authored-by: Etan Kissling <etan@status.im>
Co-authored-by: tersec <tersec@users.noreply.github.com>
2022-06-15 02:38:27 +00:00
Etan Kissling 15967c4076
keep track of latest blocks for optimistic sync (#3715)
When launched with `--light-client-enable` the latest blocks are fetched
and optimistic candidate blocks are passed to a callback (log for now).
This helps accelerate syncing in the future (optimistic sync).
2022-06-10 14:16:37 +00:00
Etan Kissling 72a46bd520
integrate light client into beacon node (#3557)
Adds a `LightClient` instance to the beacon node as preparation to
accelerate syncing in the future (optimistic sync).

- `--light-client-enable` turns on the feature
- `--light-client-trusted-block-root` configures block to start from

If no block root is configured, light client tracks DAG `finalizedHead`.
2022-06-07 19:01:11 +02:00
Etan Kissling 01efa93cf6
add light client (standalone) (#3653)
Introduces a new library for syncing using libp2p based light client
sync protocol, and adds a new `nimbus_light_client` executable that uses
this library for syncing. The new executable emits log messages when
new beacon block headers are received, and is integrated into testing.
2022-05-31 12:45:37 +02:00
tersec d35d408fa7
remove deprecated JSON-RPC server (#3656)
* remove deprecated JSON-RPC server

* keep the command-line options around as no-ops, temporarily

* service -> server; JSON-RPC is still used elsewhere
2022-05-24 07:23:48 +00:00
Etan Kissling c808f17a37
update to latest light client libp2p protocol (#3623)
Incorporates the latest changes to the light client sync protocol based
on Devconnect AMS feedback. Note that this breaks compatibility with the
previous prototype, due to changes to data structures and endpoints.
See https://github.com/ethereum/consensus-specs/pull/2802
2022-05-23 14:02:54 +02:00
Jacek Sieka f70ff38b53
enable `styleCheck:usages` (#3573)
Some upstream repos still need fixes, but this gets us close enough that
style hints can be enabled by default.

In general, "canonical" spellings are preferred even if they violate
nep-1 - this applies in particular to spec-related stuff like
`genesis_validators_root` which appears throughout the codebase.
2022-04-08 16:22:49 +00:00
Jacek Sieka 1760f4d7a7
move wallet/deposit commands to separate files (#3372)
These commands have little to do with the "normal" beacon node operation
- ergo, they deserve to live in their own module.

* clean up imports/exports
2022-02-11 21:40:49 +01:00
Jacek Sieka 49282e9477
val_mon: register locally produced aggregates (#3352)
These use a separate flow, and were previously only registered from the
network

* don't log successes in totals mode (TMI)
* remove `attestation-sent` event which is unused
2022-02-04 08:33:20 +01:00
Jacek Sieka 570379d3d9
Backfiller (#3263)
Backfilling is the process of downloading historical blocks via P2P that
are required to fulfill `GetBlocksByRange` duties - this happens during
both trusted node and finalized checkpoint syncs.

In particular, backfilling happens after syncing to head, such that
attestation work can start as soon as possible.

* Fix SyncQueue initialization procedure.
Remove usage of `awaitne`.
Add cancellation support.
Remove unneeded `sleepAsync()` if peer's head is older than needed.
Add `direction` field to all logs.
Fix syncmanager wedge issue.
Add proper resource cleaning procedure on backward sync finish.

Co-authored-by: cheatfate <eugene.kabanov@status.im>
2022-01-20 08:25:45 +01:00
zah fba1f08a5e
Implement #3129 (Optimized history traversals in the REST API) (#3219)
* Fix REST some rest call signatures and implement a simple API benchmark tool

* Implement #3129 (Optimized history traversals in the REST API)

Other notable changes:

The `updateStateData` procedure in the `blockchain_dag.nim` module is
optimized to not rewind down to the last snapshot state saved in the
database if the supplied input state can be used as a starting point
instead.

* Disallow await in withStateForBlockSlot
2022-01-05 15:49:10 +01:00