Commit Graph

589 Commits

Author SHA1 Message Date
Eugene Kabanov f088e5f57b
Consensus block value calculation for produceBlockV3 API call. (#5873)
* allow specifying get_proposer_reward block root at state.slot

* Add consensus_block_value calculation.

* Address review comments.

* Post-rebase adjustments.

* Use proper state to calculate consensus block value.

* Revert "allow specifying get_proposer_reward block root at state.slot"

This reverts commit 9fef9a8199f63056060527ac2531acc3b0ed8dcb.

* Fix post-revert problems.
Return back to Gwei.

* Adding test which is not working.

* Do not use test suite if it does not have post-state.

* Add debug logging.

* Increase logging to track sources of balance changes.

* Fix sync committee rewards/penalties calculation.

* Revert "Increase logging to track sources of balance changes."

This reverts commit 32feb20f2fdb66521401710866cd59ecc9951ef8.

* Adopt new vision to block rewards.

* Add block produce logging to VC.

* Remove rewards.nim.

* Eliminate toWei changes.

* Improve UInt256 shortLog.

* Fix conversion procedure.

* Address review comments.

* Fix test.

* Revert "Fix test."

This reverts commit 4948b2c1ec.

---------

Co-authored-by: tersec <tersec@users.noreply.github.com>
Co-authored-by: Etan Kissling <etan@status.im>
2024-03-11 14:18:50 +00:00
tersec b4db011daf
enable --styleCheck:error in ncli/ and research/ (#6014) 2024-03-02 06:25:17 +01:00
tersec e12f5e0bba
rm more Capella builder API remote signer support (#6012) 2024-03-01 22:29:47 +00:00
tersec 84034c0379
rm Capella builder API-related remote signer support (#6003) 2024-03-01 05:30:09 +00:00
tersec 684de046db
switch Builder API validator registration error to warning (#6005) 2024-03-01 06:25:29 +01:00
tersec f076502e25
rm Capella builder API bid types and blinded block construction (#6002) 2024-03-01 00:02:13 +00:00
tersec 5da2bcd243
rm Capella builder API REST calls (#5997) 2024-02-29 12:37:08 +00:00
tersec 84b752c7a1
rm REST blinded forked Capella block support (#5994) 2024-02-28 18:27:26 +00:00
Etan Kissling 4e9bc7f570
add EIP-7044 support to keymanager API (#5959)
* add EIP-7044 support to keymanager API

When trying to sign `VoluntaryExit` via keymanager API, the logic is not
yet aware of EIP-7044 (part of Deneb). This patch adds missing EIP-7044
support to the keymanager API as well.

As part of this, the VC needs to become aware about:

- `CAPELLA_FORK_VERSION`: To correctly form the EIP-7044 signing domain.
  The fork schedule does not indicate which of the results, if any,
  corresponds to Capella.
- `CAPELLA_FORK_EPOCH`: To detect whether Capella was scheduled.
  If a BN does not have it in its config while other BNs have it,
  this leads to a log if Capella has not activated yet, or marks the BN
  as incompatible if Capella already activated.
- `DENEB_FORK_EPOCH`: To check whether EIP-7044 logic should be used.

Related PRs:

- #5120 added support for processing EIP-7044 `VoluntaryExit` messages
  as part of the state transition functions (tested by EF spec tests).
- #5953 synced the support from #5120 to gossip validation.
- #5954 added support to the `nimbus_beacon_node deposits exit` command.
- #5956 contains an alternative generic version of `VCForkConfig`.

* address reviewer feedback: letter case, module location, double lookup

---------

Co-authored-by: cheatfate <eugene.kabanov@status.im>

* Update beacon_chain/rpc/rest_constants.nim

* move `VCRuntimeConfig` back to `rest_types`

---------

Co-authored-by: cheatfate <eugene.kabanov@status.im>

* fix `getForkVersion` helper

---------

Co-authored-by: cheatfate <eugene.kabanov@status.im>
2024-02-26 09:48:07 +01:00
Etan Kissling acb1eb1ac6
extend notes on random Jenkins aarch64 test failures (#5962)
The weird `let` bug from #5757 appeared again :-) Document findings.
2024-02-26 03:02:03 +01:00
tersec a4f4a35845
Revert "initial Electra support skeleton" (#5955)
* Revert "initial Electra support skeleton (#5946)"

This reverts commit d09bf3b587.

* Update test_signing_node.nim
2024-02-25 19:42:44 +00:00
tersec d09bf3b587
initial Electra support skeleton (#5946) 2024-02-24 13:44:15 +00:00
tersec e865817d44
add some push raises, ElectraBeaconStateNoImmutableValidators, and ELECTRA_FORK_VERSION/EPOCH (#5949) 2024-02-24 05:08:22 +00:00
Etan Kissling a7b9efe4d6
remove MIRACL Core support (#5941)
EIP-4844 requires BLST via `nim-kzg4844`; MIRACL Core is not supported.
Therefore, remove support for the MIRACL Core backend.

- https://github.com/status-im/nim-blscurve/pull/170
2024-02-22 16:13:31 +00:00
tersec c73d7c6f6f
automated consensus spec URL updating to v1.4.0-beta.7 (#5942) 2024-02-21 19:44:48 +00:00
Jacek Sieka 8d465a7d8c
vmon: Missed block metric (#5913)
Validator monitoring gained 2 new metrics for tracking when blocks are
included or not on the head chain.

Similar to attestations, if the block is produced in epoch N, reporting
will use the state when switching to epoch N+2 to do the reporting (so
as to reasonably stabilise the block inclusion in the face of reorgs).
2024-02-20 06:40:18 +02:00
Etan Kissling 4fc1550d0f
add `{.push raises: [].}` to recently modified files (#5908)
Status Nim style mandates `{.push raises: []}.` at start of modules.
Ensure that's the case so that exceptions are properly tracked.

- https://status-im.github.io/nim-style-guide/errors.exceptions.html
- https://github.com/status-im/nim-eth/pull/614#discussion_r1220906149
2024-02-18 01:16:49 +00:00
Jacek Sieka b5089ebf70
log elmanager timeouts (#5895)
Also:

* remove some unused metrics
* simplify execution payload fetching flow
2024-02-17 10:15:02 +01:00
tersec 8240c1bf34
use decimal representations of engine and builder bid values (#5879) 2024-02-10 05:13:00 +01:00
Etan Kissling 9593ef74b8
do not cache zero block hash if block unavailable (#5865)
With checkpoint sync, the checkpoint block is typically unavailable at
the start, and only backfilled later. To avoid treating it as having
zero hash, execution disabled in some contexts, wrap the result of
`loadExecutionBlockHash` in `Opt` and handle block hash being unknown.

---------

Co-authored-by: Jacek Sieka <jacek@status.im>
2024-02-09 22:10:38 +00:00
Etan Kissling e398078abc
`...ExecutionPayloadHash` --> `...ExecutionBlockHash` (#5864)
Finish the rename started in #4809 to have a consistent naming.
`ExecutionPayloadHash` suggests hash over payload instead of block.
`BlockHash` is also the canonical name in engine API.
2024-02-08 01:24:55 +01:00
Jacek Sieka 47704bde14
raises for beacon validators & router (#5826)
Changes here are more significant because of some good old tech debt in
block production which has grown quite hairy - the reduction in
exception handling at least provides some steps in the right direction.
2024-02-07 12:26:04 +01:00
Eugene Kabanov 21efe7e060
VC: Use produceBlockV3 when its available. (#5842)
* Initial commit.

* Add helper functions and publishBlock() implementations.

* Address review comments.
2024-02-02 15:24:40 +00:00
tersec 0638741f8b
halve validator registration chunk size (#5837) 2024-01-29 14:09:09 +01:00
tersec 128834a8eb
use `RestPlainResponse` to improve builder API rerror reporting (#5819) 2024-01-24 23:27:22 +00:00
tersec d8a2690a92
update builder API spec reference URLs to v0.4.0 (#5812) 2024-01-22 08:36:46 +01:00
tersec 4ec36e0670
Revert "use `RestPlainResponse` to improve builder API rerror reporting" (#5811)
* Revert "use `RestPlainResponse` to improve builder API rerror reporting"

* Update rest_deneb_mev_calls.nim

copyright year linting

* Update rest_capella_mev_calls.nim

more copyright year linting
2024-01-21 22:39:45 +00:00
tersec 6c53dc1e11
automated consensus spec URL updating to v1.4.0-beta.6 (#5804) 2024-01-20 11:19:47 +00:00
Eugene Kabanov 3648df7d4c
Fix VC not always be able to obtain feeRecipient value. (#5781)
Use state's validator value to obtain feeRecipient value.
Make feeRecipient and gasLimit calculation equal for BN and VC.
2024-01-19 14:36:04 +00:00
Etan Kissling d59632acd0
remove obsolete `curSlot` variable (#5786)
#5773 removed catching up on validator duties after lag. The `curSlot`
variable that was used originally to track catch-up progress no longer
has a use and is also no longer properly updated. Remove it.
2024-01-19 03:21:38 +00:00
tersec 545fb17649
use RestPlainResponse to improve builder API rerror reporting (#5777) 2024-01-19 03:20:47 +00:00
tersec db7909c1fe
don't catch up on validator duties (#5773) 2024-01-18 15:56:43 +00:00
Jacek Sieka d5785677a8
split out eth1chain into its own module (#5768)
reduces import junk in some places - more could be done here
2024-01-17 15:26:16 +01:00
tersec cf1bec7670
update some deprecated stew/results to results imports (#5743) 2024-01-16 22:37:14 +00:00
Etan Kissling b382833f43
workaround random `SIGSEGV` on macOS aarch64 CI (#5757)
Separate a `let` block into multiple `let` statements to reduce
probability of hitting random `SIGSEGV` during flaky CI tests.

whatever... 🤯
2024-01-16 13:41:49 +01:00
tersec 3541cfe020
remove extraneous length checks in KZG batch proofs (#5744)
* remove extraneous length checks in KZG batch proofs

* re-add winservice import but only for Windows, to avoid UnusedImport warning

* also uses establishWindowsService
2024-01-15 16:53:34 +01:00
Eugene Kabanov 5404178a40
Dissect Windows specific code from beacon node. (#5612)
* Make some startup procedures async.
Add more handful makeBannerAndConfig().

* Dissect windows service code from `nimbus_beacon_node.nim`.

* Add report service startup errors using windows error codes.
Add plug able exitService().

Co-authored-by: Zahary Karadjov <zahary@status.im>
Co-authored-by: Jacek Sieka <jacek@status.im>
2024-01-13 12:53:53 +02:00
Jacek Sieka b98f46c04d
Avoid global in p2p macro (fixes #4578) (#5719)
* Avoid global in p2p macro (fixes #4578)

* copy p2p macro to this repo and start de-crufting it
* make protocol registration dynamic, removing light client hacks et al
* split out light client protocol into its own file

* cleanups

* Option -> Opt
* remove more cruft

* further split beacon_sync

this allows the light client to respond to peer metadata messages
without exposing the block sync protocol

* better protocol init

* "constant" protocol index

* avoid casts

* copyright

* move some discovery code to discovery

* avoid extraneous data copy when sending chunks

* remove redundant forkdigest field

* document how to connect to a specific peer
2024-01-13 11:54:24 +02:00
tersec 251143fd51
attest to known valid block when possible (#5313)
* attest to known valid block when possible

* cleaner approach; slot is always == attestation slot itself

* copyright year linting
2024-01-11 22:34:10 +00:00
Jacek Sieka 62cbdeefc5
verify `genesis_time` more strictly (fixes #1667) (#5694)
Bogus values lead to crashes down the line when timers overflow
2024-01-06 15:26:56 +01:00
Etan Kissling 030226148d
rename `exit_pool` > `validator_change_pool` (#5679)
The `ExitPool` was renamed to `ValidatorChangePool` with Capella, but
the files were still using the previous name. Rename for consistency.
2023-12-23 06:55:47 +01:00
tersec 9c6ba7d142
consensus spec v1.4.0-beta.5 URL updates (#5672) 2023-12-16 03:27:06 +01:00
tersec cb6b54ec89
log engine/builder API decisionmaking (#5608) 2023-12-15 22:31:14 +02:00
andri lim 15147cccb1
Bump nim-web3 to dcabb8f29ee55afedefdf93cd3e102bb1daee354 (#5664)
* bump nim-web3 to dcabb8f29ee55afedefdf93cd3e102bb1daee354

also bump json-rpc to a8731e91bc336d930ac66f985d3b88ed7cf2a7d7
2023-12-12 22:15:00 +07:00
tersec 4776fecc33
consensus spec v1.4.0-beta.5 URL updates (#5655) 2023-12-06 22:16:55 +00:00
tersec 91029ce6d6
fix XDeclaredButNotUsed hints (#5652) 2023-12-06 17:23:45 +01:00
tersec c36d2aa103
fix XDeclaredButNotUsed warnings (#5648) 2023-12-05 11:45:47 +00:00
tersec 9efb2958ec
automated consensus spec URL updating to v1.4.0-beta.5 (#5647) 2023-12-05 03:34:45 +01:00
Etan Kissling b0839d1ae5
use correct KZG commitments in Deneb `constructPlainBlindedBlock` (#5642)
For Deneb, extend on #5639 and use correct KZG commitments when
producing new blinded blocks using Nimbus VC.
2023-12-04 17:36:50 +01:00
tersec 144d453f4a
Update to current (deprecated, but) version of produceBlindedBlock (#5639) 2023-12-03 10:04:12 +01:00
tersec 2fc43c9ba7
track block/blob matching/quarantines using both indices and commitments (#5621) 2023-12-01 18:58:46 +00:00
Eugene Kabanov e2e4912645
REST API produceBlockV3 implementation (#5474)
Co-authored-by: Etan Kissling <etan@status.im>
Co-authored-by: Jacek Sieka <jacek@status.im>
2023-11-29 00:30:14 +01:00
tersec ab5343d1bc
update some consensus spec URLs to v1.4.0-beta.4 (#5631) 2023-11-27 19:56:34 +01:00
tersec 0e5234efcc
avoid potentially subtle template/function symbol name interactions (#5622)
* avoid potentially subtle template/function symbol name interactions

* use warn instead of error in getExecutionPayload codepath to ensure lack of ambiguity
2023-11-24 16:34:25 +00:00
Jacek Sieka e1e809eeb7
Batch slashing protection registration (#5604)
This PR brings down the time to send 100 attestations from ~1s to
~100ms, making it feasible to run 10k validators on a single node (which
regularly send 300 attestations / slot).

This is done by batching the slashing protection database write in a
single transaction thus avoiding a slow fsync for every signature -
effects will be more pronounced on slow drives.

The benefit applies both to beacon and client validators.
2023-11-19 14:08:07 +01:00
stephanep dfd7b9bc61
Fix handling of VC validators by per payload-builder payload builder validator registration (issue #5599). (#5603)
The proposed change ensures that VC validators are registered with the builder specified by the `--payload-builder-url` argument even if the beacon node has no attached validators. It also prevent such validators from being unintentionally registered with builders configured for specific attached validators by the keymanager api.
Per the current specs, the VC have no way to specify which builders the BN should use on a per-node basis, so for the time being we have to resort to using the BN fallback default builder URL for VC validators.
2023-11-17 17:42:00 +03:00
Etan Kissling b9a693d5fe
send LC finality update on event stream on supermajority (#5602)
When new finality is reached without supermajority sync committee
support, trigger another event push on beacon-API and libp2p once
the finality gains supermajority support.

- https://github.com/ethereum/consensus-specs/pull/3549
2023-11-16 19:57:15 -08:00
Etan Kissling 98e969084d
update Deneb for latest builder-specs flow (#5598)
The `BlobSidecar` construction has been moved to the relay and is no
longer done by the BN / VC in blinded flow. Builder bid contents have
been shrinked from full `BlindedBlobBundle` to `blob_kzg_commitments`.

- https://github.com/ethereum/builder-specs/pull/90
- https://github.com/ethereum/beacon-APIs/pull/369
2023-11-15 16:20:13 -08:00
tersec c96163dbca
update some consensus spec URLs to v1.4.0-beta.4 (#5594) 2023-11-11 06:27:53 +01:00
Etan Kissling d8a7f0df81
update Deneb for blob sidecar inclusion proofs (#5565)
`BlobSidecar` is no longer signed, instead use Merkle proof to link
blobs with block.

- https://github.com/ethereum/consensus-specs/pull/3531

Associated beacon-API / builder-specs still TBD; minimal changes done
to compile in similar style to previous spec, but not standardized yet.

- https://github.com/ethereum/beacon-APIs/pull/369
- https://github.com/ethereum/builder-specs/pull/90
2023-11-06 07:48:43 +01:00
Etan Kissling a05278e263
wrap `kzgs`/`proofs`/`blobs` fields as `BlobsBundle` (#5562)
Less type conversion / copying by keeping the `BlobsBundle` together.
2023-11-04 13:49:58 +00:00
Eugene Kabanov 8cec3af61c
VC: Obtain randao signature before slot proposal. (#5490)
* Randao calculation caching for VC implementation.

* Add time monitoring for randao signatures process.

* Add delay calculation.

* Address review comments.

* Address review comments.
2023-11-04 08:14:14 +01:00
tersec ab89e96265
fill in transactions root and KZG commitments from deneb.BlindedBeaconBlock in state transition (#5553) 2023-11-02 04:56:04 +01:00
tersec 657de2721a
fill blindedblockbody KZG commitments from deneb blinded header (#5550) 2023-11-01 14:00:21 +01:00
tersec 556d5e7114
rm unused code (#5538) 2023-11-01 05:53:09 +01:00
Jacek Sieka 8c81515bf6
log slashing protection database error on failure to load (#5542)
* log slashing protection database error on failure to load

...and fix noreturn warning

* fix the copied one too

* oops

* oops 2
2023-10-31 11:15:38 +01:00
tersec 62d59daaa7
consensus-spec URL updates to v1.4.0-beta.3 (#5541) 2023-10-30 06:44:43 +00:00
tersec 34eade3a99
sign blinded blob sidecars during builder API proposals (#5537) 2023-10-28 05:10:23 +00:00
tersec b2e81de6db
remove unnecessary Builder API status endpoint usage (#5518) 2023-10-23 23:01:29 +00:00
tersec 769316d909
construct Deneb blinded blobs from builder API getHeader (#5513) 2023-10-19 17:34:32 +00:00
tersec 4ddd771127
automated consensus spec URL updating to v1.4.0-beta.3 (#5514) 2023-10-19 10:26:38 +00:00
tersec 6eb7d481fa
fix Deneb builder API block body type; introduce and use ExecutionPayloadHeaderAndBlindedBlobsBundle (#5509) 2023-10-18 02:54:02 +00:00
zah 35bf03a3fb
Add the --verifying-web3-signer-url configuration option (#5504) 2023-10-13 15:42:00 +03:00
tersec c5028b56bf
change blinded block proposal failures to REST internal errors (#5501) 2023-10-12 23:54:24 +02:00
tersec eeac00d08e
add RPC signature; add type constraints on proposeBlockMEV (#5498) 2023-10-12 11:49:48 +00:00
tersec 40616b6243
update some consensus spec URLs to v1.4.0-beta.2 (#5497) 2023-10-12 05:07:41 +00:00
tersec ec1d98cc3a
implement deneb block and blob unblinding (#5496) 2023-10-12 03:40:15 +00:00
tersec edc5c03230
consistently use SignedBlindedBeaconBlockContents; remove more Bellatrix Builder API remnants (#5493) 2023-10-11 11:48:35 +00:00
tersec 447786518f
ShufflingRef approach to next-epoch validator duty calculation/prediction (#5414)
* ShufflingRef approach to next-epoch validator duty calculation/prediction

* refactor action_tracker.updateActions to take ShufflingRef + beacon_proposers; refactor maybeUpdateActionTrackerNextEpoch to be separate and reused function; add actual fallback logic

* document one possible set of conditions

* check epoch participation flags and inactivity scores to ensure no penalties and MAX_EFFECTIVE_BALANCE to ensure rewards don't matter

* correctly (un)shuffle each proposer index

* remove debugging assertion
2023-10-10 00:02:07 +00:00
Eugene Kabanov 1da202156f
Improve keystore loading log statements to show information about why keystore was not loaded. (#5480)
Fix checkKeyName issue for `00` case.
2023-10-05 17:26:46 +03:00
Etan Kissling 37967ba03b
`ProveField` cleanups in `slashing_protection_common` (#5477)
Cleanup of `ProveField` warnings in `slashing_protection_common` module.
Note that `ProveField` is disabled by default in makefile, but sometimes
these pop up when doing a regular `nim c`, and cleaning these may allow
enabling the warning in some future.

As the `case` is over `a.kind`, `ProveField` warnings for `b` have to be
suppressed using `{.push.}` / `{.pop.}` for comparison operators.
2023-10-04 10:26:52 +02:00
Jacek Sieka 9a48e2e922
fix stale activity log (#5465) 2023-09-30 13:31:08 +00:00
tersec 7ecc1096e3
document strange-looking capella/bellatrix apparent mismatch in block building (#5461) 2023-09-27 16:06:22 +00:00
Etan Kissling 7c45b8f98d
convert remaining `toFork` -> `kind` for consistency (#5462)
We currently have multiple ways to obtain `ConsensusFork` or
`LcDataFork` from a forky type. Rename `toFork` to `kind`
for a consistent API naming.
2023-09-27 15:10:28 +00:00
Eugene Kabanov 88565e6da4
Make KeystoreCache working with KeystoreManagement API. (#5443) 2023-09-21 20:07:02 +02:00
Etan Kissling e7bc41e005
`blck` --> `forkyBlck` when using `withBlck` / `withStateAndBlck` (#5451)
For symmetry with `forkyState` when using `withState`, and to avoid
problems with shadowing of `blck` when using `withBlck` in `template`,
also rename the injected `blck` to `forkyBlck`.

- https://github.com/nim-lang/Nim/issues/22698
2023-09-21 12:49:14 +02:00
tersec 351472a2d4
per payload-builder payload builder validator registration (#5431) 2023-09-20 17:00:37 +00:00
tersec 325d5a6afa
log builder unblinding requests (#5434) 2023-09-15 09:27:38 +00:00
tersec dae4beb4a2
log engine getPayload requests (#5432) 2023-09-15 08:56:32 +00:00
tersec 2b4f987c80
remove pre-v1.4.0 attestation stability subnets (#5402)
* remove pre-v1.4.0 attestation stability subnets

* re-add most of should register stability subnets on attester duties test
2023-09-11 16:03:34 +00:00
zah 2b5bd74e15
Rename --validator-source to --web3-signer-url and document it (#5389)
Also allows multiple instances to be configured
2023-09-06 22:04:10 +03:00
tersec 0987628c0b
simplify beacon validator sync committee production for blocks slightly (#5364) 2023-09-05 12:31:33 +02:00
Eugene Kabanov afb5fdae34
Dynamic validators loop (#5385)
* Add periodic dynamic validators polling.
* Add dynamic validators tests.
2023-09-04 22:14:58 +03:00
tersec a8c56b1660
update some consensus-spec URLs to v1.4.0-beta.1 (#5379) 2023-09-01 09:31:52 +00:00
Eugene Kabanov 757328372a
Dynamic validators set. (#5366)
* Initial commit.

* Fix argument to be optional.

* Adopt options.md.
2023-08-31 15:16:15 +03:00
tersec db6f4e8090
update some consensus-spec URLs to v1.4.0-beta.1 (#5357) 2023-08-25 15:58:44 +00:00
Etan Kissling c211a3849e
remove `{.raises: [Defect].}` Nim 1.2 compatibility (#5352)
In Nim 1.6, `{.raises: [Defect].}` is no longer necessary. Remove it.
2023-08-25 11:29:07 +02:00
tersec b11443291b
refactor BN engine/builder block selection to work as REST block v3 backend (#5345) 2023-08-24 02:02:30 +00:00
Jacek Sieka a5151bc546
`validator_duties`->`beacon_validators` (#5333)
This PR renames the existing `validator_duties` to `beacon_validators`
and in doing so, names validators running inside the beacon node process
"beacon validators" while those running the VC can be referred to as
"client validators" to disambiguate the two.

The existing `validator_duties` instead takes on a new responsibility:
as a home for logic shared between beacon and client validators - ie
code that provides consistency in implementation and behavior between
the two modes of operation.

Not only does this simplify reasoning about where to put code -it also
reduces the number of dependencies the validator client has from ~5000
to ~3000 modules (!) according to `nim genDepend` significantly reducing
compile times.
2023-08-23 19:39:57 +03:00
Etan Kissling 2dae8ea26e
cleanup `{.raises.}` for `writeValue` (#5308)
- Remove unnecessary `Defect` references
- Remove spurious `SerializationError` references
- Remove duplicate `writeValue` template in `keystore.nim`;
  same implementation already exists a bit further above in same file.
2023-08-19 17:11:56 +02:00