Commit Graph

3280 Commits

Author SHA1 Message Date
tersec 44b38ef137
Implement /eth/v1/validator/{beacon,sync}_committee_selections Beacon API end-points (#4760) 2023-03-22 01:47:36 +02:00
zah eb678be0ff
Don't report an 'EL connection degraded' warning when the EL is not synced (#4761) 2023-03-22 00:19:36 +02:00
zah b8f2022668
Remove an overly agressive assertion (#4759) 2023-03-21 23:35:22 +02:00
Zahary Karadjov 813fddee8c
Capella hard-fork epoch for Mainnet 2023-03-21 23:33:06 +02:00
Zahary Karadjov 865c84cc18
Some documentation for the new 'deposit exit' work-around 2023-03-21 23:06:57 +02:00
cheatfate 596006be08
Workaround for issue #4216. 2023-03-21 23:06:57 +02:00
Tanguy 769ed00203
Add light client gossip metrics (#4745) 2023-03-21 08:55:48 +01:00
tersec 2f634c10a4
automated consensus spec URL updating from v1.3.0-rc.4 to rc.5 (#4756) 2023-03-21 00:42:22 +00:00
tersec 82bf949561
use consensus spec v1.3.0-rc.5 test vectors (#4754) 2023-03-20 20:17:25 +00:00
Zahary Karadjov 4d1b2dd9f5
Merge branch 'stable' into unstable 2023-03-17 17:51:39 +02:00
tersec 0c62950de6
more consensus spec v1.3.0-rc.4 URL updates (#4744) 2023-03-17 12:14:08 +00:00
tersec ec77116414
automated consensus spec URL updating from v1.3.0-rc.3 to rc.4 (#4742) 2023-03-17 01:10:31 +00:00
Etan Kissling fb26e19de9
use v1.3.0-rc.4 consensus spec test vectors (#4740) 2023-03-16 21:35:20 +01:00
Zahary Karadjov 3615606914
Version 23.3.1 2023-03-14 19:43:11 +02:00
Zahary Karadjov ee610cbf34
Restore the hidden --web3-force-polling option as a deprecated one 2023-03-14 18:54:15 +02:00
Zahary Karadjov 957a8c2fdc
Bugfix: A single unsynced EL client may cause an inappropriate switch to optimistic mode. 2023-03-14 18:19:11 +02:00
Zahary Karadjov d899a6a834
Prevent a potential panic on dereferencing a 'none' value 2023-03-14 18:05:29 +02:00
Zahary Karadjov 46f48269ef
Backwards compatible handling of the web3-url parameter in TOML 2023-03-14 17:50:03 +02:00
Zahary Karadjov 3a35809a02
Backwards-compatible handling of Engine URLs that don't include a specified protocol 2023-03-14 14:02:50 +02:00
henridf 9906a0d5a2
Gate Deneb `beacon_block` subscription (#4735) 2023-03-14 11:52:01 +00:00
Zahary Karadjov 2ee15a38ed
Reduce the number of block timestamp requests
The timestamps are only relevant within the candidate block range.
2023-03-14 13:47:13 +02:00
Zahary Karadjov 6fb48aca7d
Lower the maximum block range in getLogs requests to 1000 2023-03-14 13:45:01 +02:00
Etan Kissling ceefc49965
rm redundant `ConsensusFork.toString` (#4733) 2023-03-14 01:24:10 +00:00
Etan Kissling 5ef1ed5d96
indicate Capella / Shanghai readiness on startup (#4630)
In startup "Launching beacon node" message, log readiness for Capella /
Shanghai as part of config to aid with support requests.
2023-03-13 22:01:15 +01:00
Etan Kissling 29a699f69d
rename `blockFork` > `consensusFork` (#4731)
* rename `blockFork` > `consensusFork`

* revert one overzealous stateFork / blockFork rename
2023-03-12 18:48:38 +00:00
Etan Kissling f0dcbaacff
extend `addTestBlock` with Capella/Deneb support (#4728)
* extend `addTestBlock` with Capella/Deneb support

* bump copyright years

* fix
2023-03-11 20:11:33 +00:00
Etan Kissling fd8e86972d
rename `lcDataForkAtStateFork` > `lcDataForkAtConsensusFork` (#4726) 2023-03-11 20:09:21 +00:00
Etan Kissling 3e872bec86
rename `eip4844_preset.nim` > `deneb_preset.nim` (#4724) 2023-03-11 17:00:35 +00:00
Etan Kissling d8f39dd24f
rename `stateForkForDigest` > `consensusForkForDigest` (#4727) 2023-03-11 16:58:48 +00:00
Etan Kissling b0cccae7f4
rename `(with|at)StateFork` > `(with|at)ConsensusFork` (#4725) 2023-03-11 14:39:29 +00:00
Etan Kissling eba5f9af26
rename `upgrade_to_eip4844` > `upgrade_to_deneb` (#4723) 2023-03-11 14:28:55 +00:00
Etan Kissling e542d079b0
rename Deneb database tables (not in testnets atm) (#4720) 2023-03-11 02:26:03 +01:00
Etan Kissling 69013d153c
bump light client spec references to `v1.3.0-rc.3` (#4719) 2023-03-11 01:11:51 +00:00
Zahary Karadjov 17c0eeeede
Version 23.3.0 2023-03-11 02:51:44 +02:00
Etan Kissling ad118cd354
rename `stateFork` > `consensusFork` (#4718)
Just the variable, not yet `lcDataForkAtStateFork` / `atStateFork`.

- Shorten comment in `light_client.nim` to keep line width
- Do not rename `stateFork` mention in `runProposalForkchoiceUpdated`.
- Do not rename `stateFork` in `getStateField(dag.headState, fork)`

Rest is just a mechanical mass replace
2023-03-11 00:35:52 +00:00
Etan Kissling 969c6f73ae
misc local `EIP4844` > `Deneb` bumps (#4717)
* misc local `EIP4844` > `Deneb` bumps

* fix
2023-03-11 00:28:19 +00:00
Zahary Karadjov f91fe5646e
Redo fix #4588 2023-03-10 19:40:27 +02:00
Zahary Karadjov 89dccedcc9
Prevent the build up of parallel 'waitELToSync' operations 2023-03-10 19:24:20 +02:00
Etan Kissling 57b2151f95
rename `EIP4844` > `Deneb` for light client (#4713)
* rename `EIP4844` > `Deneb` for light client

* regenerate test logs
2023-03-10 17:16:26 +00:00
Etan Kissling 8fd3723e82
rename `EIP4844BeaconStateNoImmutableValidators` > `DenebXyz` (#4712) 2023-03-10 17:15:08 +00:00
Etan Kissling b0577bdc6c
rename `OnEIP4844BlockAdded` > `OnDenebBlockAdded` (#4714) 2023-03-10 17:14:27 +00:00
Etan Kissling e4b5478fff
rename `ForkDigests.eip4844` > `ForkDigests.deneb` (#4715) 2023-03-10 17:13:40 +00:00
henridf f5612f2a77
Remove BlobsSidecar used in BeaconChainDB (#4710) 2023-03-10 12:51:36 +00:00
Zahary Karadjov 70d3045584
Cosmetic logging changes in the EL manager 2023-03-10 11:12:29 +02:00
zah c6ce7f383d
More forceful EL connection resetting (fixes #4424) (#4708) 2023-03-09 23:41:28 +00:00
Zahary Karadjov f5a3ea6cbb
Fix an accidentally wrong partial git stage 2023-03-09 22:30:01 +02:00
Zahary Karadjov e808fda02c
engine_api_response_time renamed to engine_api_request_duration_seconds 2023-03-09 19:29:24 +02:00
zah ef20e831a8
More metrics for the EL monitor (#4707)
* `engine_api_response_time` provides a histogram for the Engine API
  response times for each unique pair ot URL and request type.

* All engine API requests are now tracked

Other changes:

The client will no longer exit on start-up if it fails to connect to
a properly configured EL node.
2023-03-09 19:29:04 +02:00
tersec 8625704dde
don't clear validator change subpools after creating each block (#4704)
* don't clear validator change subpools after creating each block

* remove now-spurious tests of removed behavior
2023-03-09 18:19:36 +02:00
tersec a47f0b054e
finish eip4844 to deneb module rename (#4705) 2023-03-09 01:34:17 +01:00
Etan Kissling 879f8ab5e8
check correct fork version in LC sync protocol (#4703)
Tests updated: https://github.com/ethereum/consensus-specs/pull/3284
2023-03-08 19:59:21 +01:00
Etan Kissling e499c709f4
avoid pruning LC data pre dag.tail.slot (#4702)
When using `--history=prune`, `dag.tail.slot` may advance beyond the
configured light client data retention period. Update the LC logic so
that the `dag.tail.slot` is no longer considered for LC pruning.
It is still considered to check whether new data can be produced.
2023-03-08 19:59:00 +01:00
henridf 90640cce05
Update sync to use post-decoupling RPC (#4701)
* Update sync to use post-decoupling RPCs

blob_sidecars_by_range returns a flat list of sidecars, which must
then be grouped per-slot.

* Add test for groupBlobs

* createBlobs: convert proc to func
2023-03-07 20:19:17 +00:00
tersec 2c0762013e
bump eth2-networks to include goerli/prater capella fork epoch (#4699) 2023-03-07 05:03:47 +00:00
zah 9cd30768e7
Bugfix: historic blocks violate the latest specs for the extraData field (fixes #4695) (#4700) 2023-03-07 04:43:27 +00:00
tersec 63b1b0840f
5 more modules of eip4844.foo to deneb.foo renames (#4698) 2023-03-06 18:45:52 +00:00
tersec 8541674498
simplify ELMonitor fcU payload attributes handling (#4696) 2023-03-06 16:19:15 +00:00
zah 8771e91d53
Support for driving multiple EL nodes from a single Nimbus BN (#4465)
* Support for driving multiple EL nodes from a single Nimbus BN

Full list of changes:

* Eth1Monitor has been renamed to ELManager to match its current
  responsibilities better.

* The ELManager is no longer optional in the code (it won't have
  a nil value under any circumstances).

* The support for subscribing for headers was removed as it only
  worked with WebSockets and contributed significant complexity
  while bringing only a very minor advantage.

* The `--web3-url` parameter has been deprecated in favor of a
  new `--el` parameter. The new parameter has a reasonable default
  value and supports specifying a different JWT for each connection.
  Each connection can also be configured with a different set of
  responsibilities (e.g. download deposits, validate blocks and/or
  produce blocks). On the command-line, these properties can be
  configured through URL properties stored in the #anchor part of
  the URL. In TOML files, they come with a very natural syntax
  (althrough the URL scheme is also supported).

* The previously scattered EL-related state and logic is now moved
  to `eth1_monitor.nim` (this module will be renamed to `el_manager.nim`
  in a follow-up commit). State is assigned properly either to the
  `ELManager` or the to individual `ELConnection` objects where
  appropriate.

  The ELManager executes all Engine API requests against all attached
  EL nodes, in parallel. It compares their results and if there is a
  disagreement regarding the validity of a certain payload, this is
  detected and the beacon node is protected from publishing a block
  with a potential execution layer consensus bug in it.

  The BN provides metrics per EL node for the number of successful or
  failed requests for each type Engine API requests. If an EL node
  goes offline and connectivity is resoted later, we report the
  problem and the remedy in edge-triggered fashion.

* More progress towards implementing Deneb block production in the VC
  and comparing the value of blocks produced by the EL and the builder
  API.

* Adds a Makefile target for the zhejiang testnet
2023-03-05 01:40:21 +00:00
tersec e3d96ef147
rename most eip4844Data to denebData (#4693) 2023-03-04 22:23:52 +00:00
tersec 3b41e6a0e7
rename ConsensusFork.EIP4844 to ConsensusFork.Deneb (#4692) 2023-03-04 13:35:39 +00:00
Eugene Kabanov c8b50765cf
Various fixes for VC and BN REST server. (#4673)
* Fix issue when VC unable to detect errors properly and act accordingly.
Switch all API functions used by VC to RestPlainResponse, this allows us to print errors returned by BN servers.

* Fix issue when prepareBeaconCommitteeSubnet() do not perform actions when BN is optimistically synced only.

* Fix Defect issue.

* Fix submit/publish returning `false` when operation was successful.

* Address review comments.

* Fix some client calls unable to receive `execution_optimistic` field, mark BN as OptSynced when such request has been made.

* Adjust warning levels.

---------

Co-authored-by: Jacek Sieka <jacek@status.im>
2023-03-03 20:20:01 +00:00
Jacek Sieka fec580c39c
doppelganger: guard against reordering (#4691) 2023-03-03 18:38:48 +01:00
Jacek Sieka 65aee9cf22
ncli_db: use fork from runtimeconfig when verifying era file (#4689)
Sepolia in particular goes through two hard forks during era 1, meaning
the phase0 fork is no longer part of the state, even though blocks that
belong to it still are phase0.
2023-03-03 13:15:11 +01:00
tersec ea060de6d4
more eip4844 -> deneb module references (#4690) 2023-03-02 21:09:24 +01:00
tersec d058aa09c8
more withdrowls (#4674) 2023-03-02 17:13:35 +01:00
Jacek Sieka 4d9eaafe9c
vc: fix missing attestations due to doppelganger in epoch 1 (#4688)
* log validator that triggers doppelganger
* move activity detection closer to where it's performed, record
aggregation as activity (since it's part of the liveness endpoint)
* vc: check for doppelgangers in epoch 0 also (so that activity in epoch
1 can happen)
* avoid some looping when processing activities
2023-03-02 16:55:45 +01:00
henridf 1de3cf5246
Remove SignedBeaconBlockAndBlobsSidecar (#4683)
This commit removes SignedBeaconBlockAndBlobsSidecar and all remaining
references.
2023-03-02 15:12:04 +01:00
tersec 88092bb411
don't try to validate execution block hashes of non-execution payloads (#4687) 2023-03-02 00:11:46 +00:00
henridf 5a9e63a029
Remove uses of beacon_block_and_blobs_sidecar topic (#4682)
* Remove use of beacon_block_and_blobs_sidecar topic

This topic goes away with decoupled blocks and blobs.

* remove use of getBeaconBlockAndBlobsSidecarTopic from test

* update nimbus_light_client.nim
2023-03-01 21:30:20 +01:00
henridf 3681177cf4
Remove ForkySignedBeaconBlockMaybeBlobs (#4681)
This commit removes ForkySignedBeaconBlockMaybeBlobs and all
references. I tried to pull that thread only as little as was needed
to get rid of it. Left a placeholder BlobSidecar array (in lieu of
Opt[BlobsSidecar]) in a few places; this will be used as we rebuild
the decoupled implementation.
2023-02-28 11:36:17 +00:00
tersec fb1b7458f0
almost all remaining automated v1.3.0-rc.2 to rc.3 consensus spec URL updates (#4679) 2023-02-27 21:37:37 +00:00
tersec 8ee5e8271d
weaken overaggressive different-fork-next-slot check (#4665) 2023-02-27 21:23:00 +01:00
henridf 0a87889eaf
Add blobSidecar RPCs (#4677)
* Add blob_sidecars_by_range RPC

* Add blob_sidecars_by_root RPC
2023-02-27 21:16:59 +01:00
henridf 144130e80e
Add DB support for BlobSidecar (#4675) 2023-02-27 14:02:37 +00:00
henridf dede36fe86
Remove blobsSidecar from orphans table (#4670) 2023-02-27 06:10:22 +00:00
henridf a999da3361
Remove beacon_block_and_blobs_sidecar_by_root RPC (#4672)
Along with its use in the request manager.
2023-02-26 23:17:40 +00:00
tersec 982d79f9a2
more eip4844 -> deneb changes (#4666) 2023-02-25 03:03:34 +02:00
tersec 2dd2a782d9
disable BareExcept warnings (#4662)
* disable BareExcept warnings

* BareExcept only exists in >= 1.6.11
2023-02-24 19:50:46 +00:00
Eugene Kabanov e21742a325
Fix attester and proposer duties optimistic execution behavior. (#4657)
* Fix attester and proposer duties optimistic execution behavior.

* Address review comments.

* Address review comments more efficiently.
2023-02-23 23:13:17 +00:00
tersec 069d973fc6
use 'deneb' in 4 more modules (#4658) 2023-02-23 19:06:57 +01:00
Eugene Kabanov dd911a7ac6
Fix IndexError for reasons. (#4656)
* Fix IndexError for reasons.

* Deprecate specific Timeout reason.
Log exception message instead of single Timeout error.
2023-02-23 17:02:17 +01:00
tersec 8f269c92d7
rename eip48844ImplementationMissing to denebImplementationMissing (#4654) 2023-02-23 10:37:45 +00:00
tersec c6cefd8914
add success logging for being configured to use payload builder (#4653) 2023-02-23 11:46:18 +02:00
zah 6036f2e7d7
Local sim impovements (#4551)
* Local sim impovements

* Added support for running Capella and EIP-4844 simulations
  by downloading the correct version of Geth.

* Added support for using Nimbus remote signer and Web3Signer.
  Use 2 out of 3 threshold signing configuration in the mainnet
  configuration and regular remote signing in the minimal one.

* The local testnet simulation can now use a payload builder.
  This is currently not activated in CI due to lack of automated
  procedures for installing third-party relays or builders.

  You are adviced to use mergemock for now, but for most realistic
  results, we can create a simple builder based on the nimbus-eth1
  codebase that will be able to propose transactions from the regular
  network mempool.

* Start the simulation from a merged state. This would allow us
  to start removing pre-merge functionality such as the gossip
  subsciption logic. The commit also removes the merge-forcing
  hack installed after the TTD removal.

* Consolidate all the tools used in the local simulation into a
  single `ncli_testnet` binary.
2023-02-23 02:10:07 +00:00
Eugene Kabanov 08b6bb7a6b
VC: Use not-synced/opt-synced BNs. (#4635)
* Initial commit.

* Address review comments and recommendations.

* Fix too often `Execution client not in sync` messages in logs.

* Add failure reason for duties requests.

* Add more reasons to every place of ValidatorApiError.

* Address race condition issue.

* Remove `vc` argument for getFailureReason().
2023-02-23 00:11:00 +00:00
tersec 1dd07d5def
switch 3 modules in beacon_chain/spec/ from eip4844 to deneb (#4651) 2023-02-22 14:10:00 +01:00
Jakub Sokołowski 1a0b2daa0a
beacon_chain: fix history flag help message (#4650)
Signed-off-by: Jakub Sokołowski <jakub@status.im>
2023-02-22 14:09:38 +01:00
tersec ffaa42b880
fix Deneb/EIP4844 phrasing (#4646) 2023-02-21 19:25:07 +01:00
tersec 29fb65a9db
automated update of v1.3.0-rc.2 to v1.3.0-rc.3 consensus spec URLs (#4647) 2023-02-21 16:43:21 +00:00
tersec 79eddcde40
capella builder API support (#4643)
* capella builder API support

* use capella EPH when appropriate

* fill in ExecutionPayload.withdrawals and sanity-check builder API withdrawals root
2023-02-21 14:21:38 +01:00
tersec 68cb9fe7b1
use consensus spec v1.3.0-rc.3 test vectors (#4645) 2023-02-21 12:58:08 +00:00
Jacek Sieka 83f9745df1
restore doppelganger check on connectivity loss (#4616)
* restore doppelganger check on connectivity loss

https://github.com/status-im/nimbus-eth2/pull/4398 introduced a
regression in functionality where doppelganger detection would not be
rerun during connectivity loss. This PR reintroduces this check and
makes some adjustments to the implementation to simplify the code flow
for both BN and VC.

* track when check was last performed for each validator (to deal with
late-added validators)
* track when we performed a doppel-detectable activity (attesting) so as
to avoid false positives
* remove nodeStart special case (this should be treated the same as
adding a validator dynamically just after startup)

* allow sync committee duties in doppelganger period

* don't trigger doppelganger when registering duties

* fix crash when expected index response is missing

* fix missing slashingSafe propagation
2023-02-20 13:28:56 +02:00
tersec b19f7001aa
log expected_withdrawals discrepancy (#4595)
* log expected_withdrawals discrepancy

* capella-only rather than not-bellatrix

* add one more (in theory unnecessary, but) guard against get(none)
2023-02-20 09:46:37 +01:00
tersec 72797cad09
switch some beacon_chain/spec/ from eip4844.Foo to deneb.Foo (#4642) 2023-02-20 09:45:49 +01:00
tersec 629b005c27
refactor batch validation not to require genesis_validators_root each time (#4640) 2023-02-20 09:26:22 +01:00
tersec 0cc0c7e6b7
builder spec v0.3.0 URL updates (#4641)
* builder spec v0.3.0 URL updates

* remove stray non-URL/comment change
2023-02-18 01:54:30 +01:00
tersec a382498cfe
batch-verify BLS to execution change messages (#4637) 2023-02-17 13:35:12 +00:00
tersec dc0bbe3a57
rm blockForkAtEpoch and switch callers to consensusForkAtEpoch (#4634) 2023-02-16 21:16:54 +01:00
Zahary Karadjov 8202d4a84e
Version 23.2.0 2023-02-16 20:19:22 +02:00
Eugene Kabanov e91415662b
Keystore cache implementation. (#4372) 2023-02-16 19:25:48 +02:00