Commit Graph

99 Commits

Author SHA1 Message Date
Jacek Sieka a5151bc546
`validator_duties`->`beacon_validators` (#5333)
This PR renames the existing `validator_duties` to `beacon_validators`
and in doing so, names validators running inside the beacon node process
"beacon validators" while those running the VC can be referred to as
"client validators" to disambiguate the two.

The existing `validator_duties` instead takes on a new responsibility:
as a home for logic shared between beacon and client validators - ie
code that provides consistency in implementation and behavior between
the two modes of operation.

Not only does this simplify reasoning about where to put code -it also
reduces the number of dependencies the validator client has from ~5000
to ~3000 modules (!) according to `nim genDepend` significantly reducing
compile times.
2023-08-23 19:39:57 +03:00
tersec d542410a17
remove more Bellatrix-only Builder API support infrastructure (#5251) 2023-08-05 00:11:51 +00:00
henridf f13f23e21b
Complete Deneb bn side for /eth/v2/validator/blocks/{slot} (#5146)
* Complete Deneb bn side for /eth/v2/validator/blocks/{slot}

* Address review feedback

* More review feedback

* Tighten up when condition

* Update beacon_chain/rpc/rest_validator_api.nim

Co-authored-by: tersec <tersec@users.noreply.github.com>

---------

Co-authored-by: tersec <tersec@users.noreply.github.com>
2023-07-11 19:35:55 +00:00
Eugene Kabanov c0e5c26da1
VC: Use SSZ encoding while processing blocks data (#4999)
* Refactor api.nim to provide more informative failure reasons.
Distinct between unexpected data and unexpected code.
Deprecate Option[T] usage.

* Fix 400 for produceBlindedBlock().
Get proper string conversion for strategy.

* Fix SSZ encoded versions of ProduceBlockResponseV2, ProduceBlockResponseV2 can be received and decoded.
Fix done() warnings.
Bump presto.

* Fix compilation error with new presto.
Use TcpNoDelay option for Web3Signer.

* Fix produceBlockV2() should provide SSZ responses too.

* Address block encoding issue.

* Fix signing test.

* Bump presto.

* Address review comments.
2023-06-14 08:04:15 +02:00
tersec e8e67ec771
allow payload builder client to be function of validator/proposer (#5015)
* allow payload builder client to be function of validator/proposer

* fileExists has side effects on Windows and only Windows

* another not-always-func
2023-06-02 11:06:33 +00:00
tersec 7a7573d7d0
fix chronos done -> completed deprecation warnings (#5016) 2023-06-01 10:04:30 +02:00
tersec d1941b670a
refactor payload builder REST client usage (#4973)
* refactor payload builder REST client usage

* change HTTP response code
2023-05-25 18:38:56 +03:00
Etan Kissling 00728e9bb7
use `executionValid` bit in `BlockRef` (#4956)
Update `beacon_node/rpc` usage for the new `executionValid` field.
2023-05-25 15:57:24 +02:00
Etan Kissling 40e89937c5
segregate sync committee messages by period / fork (#4953)
`SyncCommitteeMsgPool` grouped messages by their `beacon_block_root`.
This is problematic around sync committee period boundaries and forks.
Around sync committee period boundaries, members from both the current
and next sync committee may sign the same `beacon_block_root`; mixing
the signatures from both committees together is a mistake. Likewise,
around fork transitions, the `signing_root` changes, so those messages
also need to be segregated.
2023-05-17 07:55:55 +03:00
Jacek Sieka 83393cea8d
dependent slot helpers 2023-05-16 11:04:25 +02:00
Jacek Sieka 3acbb3ea6c
fix missing shuffling dependent slot computation 2023-05-16 10:05:35 +02:00
Etan Kissling 3a92bf3914
use dependent root as `execution_optimistic` basis for duties (#4955)
The validator beacon APIs `getAttesterDuties`, `getProposerDuties`, and
`getSyncCommitteeDuties`, have reported the `execution_optimistic`
state for the current head block. This can lead to a race if duties are
requested around the slot start, if a new head block is currently being
processed by the EL, during which the BN head may be briefly optimistic.

`execution_optimistic` is documented in beacon APIs as:

> True if the response references an unverified execution payload.
> Optimistic information may be invalidated at a later time.
> If the field is not present, assume the False value.

As the duty endpoints reference the shuffling dependent root instead of
the currently selected head block, `execution_optimistic` is now fetched
based on that shuffling dependent block root. As this dependent block is
in the past it doesn't usually become optimistic when adding new blocks.

Note that the endpoints requested 4/8 seconds into the slot that perform
the actual duties instead of just querying for duty schedule, still
report `execution_optimistic` based on the BN head block.
2023-05-15 23:42:42 +03:00
Etan Kissling 98c30f600b
fix `getSyncCommitteeDuties` for `ALTAIR_FORK_EPOCH`.period (#4954)
When fetching historical `getSyncCommitteeDuties` for the very first
sync committee period, the case must be handled where Altair may not
have been scheduled on a sync committee period boundary.
2023-05-14 23:56:50 +00:00
Etan Kissling d263f7f0cb
fix SSZ response for `produceBlindedBlock` (#4943)
* fix SSZ response for `produceBlindedBlock`

In `produceBlindedBlock`, we sent the `ForkedBlindedBeaconBlock` when
requested to reply in SSZ format. However, expected result is just the
inner `ForkyBlindedBeaconBlock` together with `eth-consensus-version`.

Note: We do not use SSZ format in our VC for this endpoint at this time,
which explains why we haven't noticed earlier.

* fix Altair/Phase0
2023-05-12 15:40:45 +00:00
tersec d3929cbb45
update some beacon API spec URLs; fix some Name and DuplicateModuleImport hints (#4929) 2023-05-10 10:20:55 +00:00
tersec 0fbf911722
use higher of available engine and builder API bids (#4795) 2023-04-11 18:19:48 +03:00
tersec 44b38ef137
Implement /eth/v1/validator/{beacon,sync}_committee_selections Beacon API end-points (#4760) 2023-03-22 01:47:36 +02:00
Etan Kissling ad118cd354
rename `stateFork` > `consensusFork` (#4718)
Just the variable, not yet `lcDataForkAtStateFork` / `atStateFork`.

- Shorten comment in `light_client.nim` to keep line width
- Do not rename `stateFork` mention in `runProposalForkchoiceUpdated`.
- Do not rename `stateFork` in `getStateField(dag.headState, fork)`

Rest is just a mechanical mass replace
2023-03-11 00:35:52 +00:00
zah 8771e91d53
Support for driving multiple EL nodes from a single Nimbus BN (#4465)
* Support for driving multiple EL nodes from a single Nimbus BN

Full list of changes:

* Eth1Monitor has been renamed to ELManager to match its current
  responsibilities better.

* The ELManager is no longer optional in the code (it won't have
  a nil value under any circumstances).

* The support for subscribing for headers was removed as it only
  worked with WebSockets and contributed significant complexity
  while bringing only a very minor advantage.

* The `--web3-url` parameter has been deprecated in favor of a
  new `--el` parameter. The new parameter has a reasonable default
  value and supports specifying a different JWT for each connection.
  Each connection can also be configured with a different set of
  responsibilities (e.g. download deposits, validate blocks and/or
  produce blocks). On the command-line, these properties can be
  configured through URL properties stored in the #anchor part of
  the URL. In TOML files, they come with a very natural syntax
  (althrough the URL scheme is also supported).

* The previously scattered EL-related state and logic is now moved
  to `eth1_monitor.nim` (this module will be renamed to `el_manager.nim`
  in a follow-up commit). State is assigned properly either to the
  `ELManager` or the to individual `ELConnection` objects where
  appropriate.

  The ELManager executes all Engine API requests against all attached
  EL nodes, in parallel. It compares their results and if there is a
  disagreement regarding the validity of a certain payload, this is
  detected and the beacon node is protected from publishing a block
  with a potential execution layer consensus bug in it.

  The BN provides metrics per EL node for the number of successful or
  failed requests for each type Engine API requests. If an EL node
  goes offline and connectivity is resoted later, we report the
  problem and the remedy in edge-triggered fashion.

* More progress towards implementing Deneb block production in the VC
  and comparing the value of blocks produced by the EL and the builder
  API.

* Adds a Makefile target for the zhejiang testnet
2023-03-05 01:40:21 +00:00
tersec 3b41e6a0e7
rename ConsensusFork.EIP4844 to ConsensusFork.Deneb (#4692) 2023-03-04 13:35:39 +00:00
Eugene Kabanov c8b50765cf
Various fixes for VC and BN REST server. (#4673)
* Fix issue when VC unable to detect errors properly and act accordingly.
Switch all API functions used by VC to RestPlainResponse, this allows us to print errors returned by BN servers.

* Fix issue when prepareBeaconCommitteeSubnet() do not perform actions when BN is optimistically synced only.

* Fix Defect issue.

* Fix submit/publish returning `false` when operation was successful.

* Address review comments.

* Fix some client calls unable to receive `execution_optimistic` field, mark BN as OptSynced when such request has been made.

* Adjust warning levels.

---------

Co-authored-by: Jacek Sieka <jacek@status.im>
2023-03-03 20:20:01 +00:00
Eugene Kabanov e21742a325
Fix attester and proposer duties optimistic execution behavior. (#4657)
* Fix attester and proposer duties optimistic execution behavior.

* Address review comments.

* Address review comments more efficiently.
2023-02-23 23:13:17 +00:00
tersec 8f269c92d7
rename eip48844ImplementationMissing to denebImplementationMissing (#4654) 2023-02-23 10:37:45 +00:00
tersec 79eddcde40
capella builder API support (#4643)
* capella builder API support

* use capella EPH when appropriate

* fill in ExecutionPayload.withdrawals and sanity-check builder API withdrawals root
2023-02-21 14:21:38 +01:00
tersec 0fb726c420
`BeaconStateFork/BeaconBlockFork` -> `ConsensusFork` (#4560)
* `BeaconStateFork/BeaconBlockFork` -> `ConsensusFork`

* revert unrelated change

* revert unrelated changes

* update test summaries
2023-01-28 19:53:41 +00:00
Jacek Sieka ba3db7aa5a
spec: Option -> Opt (#4488) 2023-01-11 12:29:21 +00:00
Etan Kissling 12b428e3d7
fix `produceBlockV2` call for Capella (#4443)
Avoid issue when calling `/eth/v2/validator/blocks/{slot}` for a
Capella slot via VC, same logic as in `validator_duties`.
2022-12-21 10:35:56 +00:00
Jacek Sieka bd8f08204e
Implement skip_randao_verification for blinded blocks (#4435)
* Implement skip_randao_verification for blinded blocks

* fix redundant randao verification on block replay (5% faster)
* check randao in REST instead of internally
* avoid redundant copies when making blocks
* cleanup leftover randao skipping code

* fix test summary
2022-12-19 15:11:12 +02:00
Jacek Sieka 6e2a02466e
unify bn/vc doppelganger detection (#4398)
* fix REST liveness endpoint responding even when gossip is not enabled
* fix VC exit code on doppelganger hit
* fix activation epoch not being updated correctly on long deposit
queues
* fix activation epoch being set incorrectly when updating validator
* move most implementation logic to `validator_pool`, add tests
* ensure consistent logging between VC and BN
* add docs
2022-12-09 17:05:55 +01:00
Eugene Kabanov a311f04a19
Implementation for REST /eth/v1/validator/liveness/{epoch} endpoint. (#4381)
Switch VC to use /eth/v1/validator/liveness/{epoch} endpoint instead of nimbus one.
2022-12-06 12:29:00 +01:00
tersec 415b11aa67
EIP4844 tweaks to pass SSZ consensus object tests (#4390) 2022-12-05 21:36:53 +00:00
tersec 4e71e77da7
structure for supporting capella block production (#4383) 2022-12-02 08:39:01 +01:00
zah d07113767d
Bugfix: The VC was producing invalid sync committee contributions (#4343)
Since the sync committee duties are no longer updated on every slot
and previously the sync committee aggregators selection proofs were
generated during the duties update, this now resulted in the client
using stale selection proofs (they must be generated at each slot).

The fix consists of moving the selection proof generation logic in
a different function which is properly executed on each slot.

Other changes:

* The logtrace tool has been enhanced with a framework for adding
  new simpler log aggregation and analysis algorithms.
  The default CI testnet simulation will now ensure that the blocks
  in the network have reasonable sync committee participation.
2022-11-24 09:46:35 +02:00
tersec ec443601eb
implement capellaImplementationMissing points; don't track not-active validator duties (#4340)
* implement several capellaImplementationMissing points

* don't register validator activity for not-active validators

* don't check validator indices already coming out of committees which exist; must be active validators, or else other deeper bugs
2022-11-22 13:56:05 +02:00
Jacek Sieka 8297b962cc
Fix VC-based sync subnet subscriptions (#4293)
* move duty tracking code to `ActionTracker`
* fix earlier duties overwriting later ones
* re-run subnet selection when new duty appears
* log upcoming duties as soon as they're known (vs 4 epochs before)
2022-11-08 12:43:38 +01:00
Eugene Kabanov 8417b7e064
Sync committee subscription fixes. (#4281)
* Preparing code.

* Fix prepareXXX procedures to use `onceToAll` strategy only.

* Remove lighthouse like subscription code.

* Address review comments.
2022-11-03 20:23:33 +01:00
tersec cee5a73a31
v1/beacon/blocks/{block_id}, v1/debug/beacon/states/{state_id}, and v1/validator/blocks/{slot} deprecated (#4279)
* v1/beacon/blocks/{block_id}, v1/debug/beacon/states/{state_id}, and v1/validator/blocks/{slot} deprecated

* Update beacon_chain/rpc/rest_constants.nim

Co-authored-by: Jacek Sieka <jacek@status.im>

* Update beacon_chain/rpc/rest_constants.nim

Co-authored-by: Jacek Sieka <jacek@status.im>

Co-authored-by: Jacek Sieka <jacek@status.im>
2022-11-02 10:56:55 +00:00
tersec 3ef09ff596
Support `/eth/v1/validator/blinded_blocks` (#4272)
* Support BN endpoints for producing blinded blocks

* use correct endpoint version

* serve either JSON or SSZ versions of endpoint
2022-10-31 18:39:03 +01:00
Jacek Sieka 1572814a0c
Highlight execution client for optimistically synced node (#4266)
...such that the user knows where to look
2022-10-27 17:22:32 +00:00
Jacek Sieka af9ec577d0
nicer error message for failed backfill (#4188)
* nicer error message for failed backfill

Many checkpoint sources don't support block download

* RestGenericError -> RestErrorMessage

...and other assorted fixes to bring rest types closer to spec

* fix tests
2022-09-29 23:55:18 +03:00
Jacek Sieka b1bc830a92
Harden EpochRef loading against bogus block root at tail (#4178)
* add more error information when things go wrong with database
* lower log level when reloading attestations from no-block epoch start
slot
2022-09-27 18:56:08 +02:00
Michael Sproul d6325b1ab5
api: add skip_randao_verification for produceBlockV2 (#3837) 2022-09-21 09:38:08 +02:00
tersec 02a99543c6
more `withState` `state` -> `forkyState` (#4112) 2022-09-13 14:53:12 +03:00
tersec 5b0b48f6e9
implement /eth/v1/validator/register_validator (#4115) 2022-09-13 14:52:26 +03:00
tersec bf3a014287
more efficient forkchoiceUpdated usage (#4055)
* more efficient forkchoiceUpdated usage

* await rather than asyncSpawn; ensure head update before dag.updateHead

* use action tracker rather than attached validators to check for next slot proposal; use wall slot + 1 rather than state slot + 1 to correctly check when missing blocks

* re-add two-fcU case for when newPayload not VALID

* check dynamicFeeRecipientsStore for potential proposal

* remove duplicate checks for whether next proposer
2022-09-07 20:34:52 +02:00
tersec 1d55743ebb
allow execution clients several seconds to construct blocks (#4012) 2022-08-23 19:19:52 +03:00
Jacek Sieka 0d9fd54857
cache shuffling separately from other EpochRef data (fixes #2677) (#3990)
In order to avoid full replays when validating attestations hailing from
untaken forks, it's better to keep shufflings separate from `EpochRef`
and perform a lookahead on the shuffling when processing the block that
determines them.

This also helps performance in the case where REST clients are trying to
perform lookahead on attestation duties and decreases memory usage by
sharing shufflings between EpochRef instances of the same dependent
root.
2022-08-18 21:07:01 +03:00
zah 06a5c67e62
The body of prepare_beacon_proposer is an array (Fixes #3920) (#3938) 2022-08-09 12:53:02 +03:00
zah cd04f27c37
Implement the /eth/v1/validator/prepare_beacon_proposer end-point (#3901) 2022-07-25 23:12:53 +03:00
Jacek Sieka e1830519a4
Introduce message router (#3829)
Whether new blocks/attestations/etc are produced internally or received
via REST, their journey through the node is the same - to ensure that
they get the same treatment (logging, metrics, processing), this PR
moves the routing to a dedicated module and fixes several small
differences that existed before.

* `xxxValidator` -> `processMessageName` - the processor also was adding
messages to pools, so we want the name to reflect that action
* add missing "sent" metrics for some messages
* document ignore policy better - already-seen messages are not actaully
rebroadcast by libp2p
* skip redundant signature checks for internal validators consistently
2022-07-06 16:11:44 +00:00