Commit Graph

3840 Commits

Author SHA1 Message Date
Etan Kissling 483e70d7a0
use `forks` sugar in `nimbus_light_client` (#5653)
Reduce code duplication and maintenance burden by using `withAll` sugar.
2023-12-06 19:44:49 +01:00
tersec 91029ce6d6
fix XDeclaredButNotUsed hints (#5652) 2023-12-06 17:23:45 +01:00
Jacek Sieka 9d155ebb35
avoid json-rpc server import (#5649) 2023-12-05 22:08:18 +01:00
tersec c36d2aa103
fix XDeclaredButNotUsed warnings (#5648) 2023-12-05 11:45:47 +00:00
Jacek Sieka 4a56faa579
era: fix verifier at empty slots (#5641)
* era: fix verifier at empty slots

* avoid returning zero-byte block data to REST/p2p when loading era
files

* fix local test
2023-12-05 07:55:25 +01:00
tersec 9efb2958ec
automated consensus spec URL updating to v1.4.0-beta.5 (#5647) 2023-12-05 03:34:45 +01:00
tersec 1791b0ee0a
fix XDeclaredButNotUsed warnings (#5638) 2023-12-04 21:14:35 +00:00
tersec 11bbc4010e
use correct pubkey for gossip validation of BLS to execution changes (#5646) 2023-12-04 22:10:13 +01:00
tersec 6a07602aca
use consensus spec test vectors v1.4.0-beta.5 (#5645) 2023-12-04 22:00:27 +01:00
Etan Kissling b0839d1ae5
use correct KZG commitments in Deneb `constructPlainBlindedBlock` (#5642)
For Deneb, extend on #5639 and use correct KZG commitments when
producing new blinded blocks using Nimbus VC.
2023-12-04 17:36:50 +01:00
tersec 144d453f4a
Update to current (deprecated, but) version of produceBlindedBlock (#5639) 2023-12-03 10:04:12 +01:00
tersec 2fc43c9ba7
track block/blob matching/quarantines using both indices and commitments (#5621) 2023-12-01 18:58:46 +00:00
tersec 6a9d522705
Apply EIP-7044 to block signature batch verification (#5637) 2023-12-01 14:44:45 +00:00
tersec 7634f92408
update network chain ID reference URL(s) (#5636) 2023-12-01 13:32:45 +00:00
tersec 91b5204e52
stop checking beacon parent root for fcU caching (#5635) 2023-12-01 12:24:23 +00:00
Eugene Kabanov 26bcb7057c
REST: Add implementation of postStateValidators and postStateValidatorBalances. (#5632)
* Add implementation POST versions of /eth/v1/beacon/states/{state_id}/validators and /eth/v1/beacon/states/{state_id}/validator_balances.
Add tests.

* Address review comments.
Fix toList() issue.

* Fix tests.

* Address review comments 2.

* Address review comments 3.
Fix unique check for validator identifiers.

* Address review comments.

* Fix constant value.
2023-11-29 12:05:03 +00:00
Etan Kissling beb915e308
remove emoji from function names (#5633)
Some editors seem to have issues with those. The ones in strings seem to
be alright, though. Rename the affected symbols.
2023-11-29 00:30:44 +00:00
Eugene Kabanov e2e4912645
REST API produceBlockV3 implementation (#5474)
Co-authored-by: Etan Kissling <etan@status.im>
Co-authored-by: Jacek Sieka <jacek@status.im>
2023-11-29 00:30:14 +01:00
Zahary Karadjov 3a527d622d
Version 23.11.0 2023-11-28 23:54:05 +02:00
tersec ab5343d1bc
update some consensus spec URLs to v1.4.0-beta.4 (#5631) 2023-11-27 19:56:34 +01:00
tersec 6dee4d59fe
switch engine authentication from `Option` to `Opt` (#5624)
* switch engine authentication from Option to Opt

* el_conf updates

* LC and el_conf tests
2023-11-27 14:48:29 +00:00
tersec 23983c34ac
rm debugRaiseAssert (#5629) 2023-11-26 19:40:10 +01:00
tersec af0b36f2e7
respond even in not-yet-supported Deneb beacon+builder API case in REST server (#5628) 2023-11-26 10:00:04 +00:00
tersec 115ffa70eb
rm unused code (#5623) 2023-11-25 12:09:18 +00:00
tersec 5aa595fbc6
Option[deneb.SignedBeaconBlock] -> Opt[deneb.SignedBeaconBlock] (#5620) 2023-11-24 17:34:45 +01:00
tersec 0e5234efcc
avoid potentially subtle template/function symbol name interactions (#5622)
* avoid potentially subtle template/function symbol name interactions

* use warn instead of error in getExecutionPayload codepath to ensure lack of ambiguity
2023-11-24 16:34:25 +00:00
Etan Kissling 8cea8af620
fix startup after BN exited between head and finalized blocks updates (#5617)
When the BN exits after writing new `head` to database, but before
completing the `updateFinalizedBlocks` call, the database is slightly
inconsistent due to the partial write. We currently fail to start up
after that. Fix that by catching up on partial `updateFinalizedBlocks`
tasks on start up, and add a test for this edge case.
2023-11-23 00:44:20 +01:00
Etan Kissling b0b8f21223
fix LC data import `--only-new` regression (#5616)
Corrects LC server startup logic for `LightClientDataImportMode.OnlyNew`
after regression in #5607.
2023-11-22 14:05:31 +01:00
Etan Kissling 7554ca2583
retry sync request when receiving invalid response range (#5615)
Inform `sync_queue` about inconsistent responses to sync requests
so that the affected requests are retried with a different peer.
2023-11-22 09:01:51 +01:00
Etan Kissling c33dd2c170
restrict best LC update collection to canonical blocks (#5613)
Simplify best `LightClientUpdate` collection by tracking only canonical
data instead of tracking the best update across all branches within the
sync committee period.

- https://github.com/ethereum/consensus-specs/pull/3553
2023-11-21 23:51:05 +01:00
Etan Kissling 946ebe54cf
simplify `chunkMaxSize[T]` helper implementation (#5614)
There is no longer a `MAX_CHUNK_SIZE_BELLATRIX` since #5284, and we have
not used separate limits since #4558. Clean up to reflect that.
2023-11-21 23:08:31 +01:00
Etan Kissling 375f276f31
reorder gossip validation checks (#5610)
* reorder gossip validation checks

Doing the coverage check only after the corresponding committee index is
known allows optimization by early rejecting invalid data.

* use same helper for individual attestations as well
2023-11-20 16:42:29 +02:00
Jacek Sieka e1e809eeb7
Batch slashing protection registration (#5604)
This PR brings down the time to send 100 attestations from ~1s to
~100ms, making it feasible to run 10k validators on a single node (which
regularly send 300 attestations / slot).

This is done by batching the slashing protection database write in a
single transaction thus avoiding a slow fsync for every signature -
effects will be more pronounced on slow drives.

The benefit applies both to beacon and client validators.
2023-11-19 14:08:07 +01:00
Etan Kissling d8144c6de1
cache new LC data after creating new LC updates (#5607)
When creating new LC updates, information about the parent block's post
state must be available (cached), but information about current block's
post state is not yet required. Caching information about the current
block's post state can be delayed, simplifying the LC data collection
logic a bit and allowing more future flexibility with the cache design.
2023-11-18 01:17:26 +00:00
Etan Kissling b179cb71a0
clean up `XDeclaredButNotUsed` for light client (#5606)
Handle remaining `XDeclaredButNotUsed` warnings for light client.
2023-11-18 00:03:27 +01:00
stephanep dfd7b9bc61
Fix handling of VC validators by per payload-builder payload builder validator registration (issue #5599). (#5603)
The proposed change ensures that VC validators are registered with the builder specified by the `--payload-builder-url` argument even if the beacon node has no attached validators. It also prevent such validators from being unintentionally registered with builders configured for specific attached validators by the keymanager api.
Per the current specs, the VC have no way to specify which builders the BN should use on a per-node basis, so for the time being we have to resort to using the BN fallback default builder URL for VC validators.
2023-11-17 17:42:00 +03:00
Etan Kissling b9a693d5fe
send LC finality update on event stream on supermajority (#5602)
When new finality is reached without supermajority sync committee
support, trigger another event push on beacon-API and libp2p once
the finality gains supermajority support.

- https://github.com/ethereum/consensus-specs/pull/3549
2023-11-16 19:57:15 -08:00
Etan Kissling 98e969084d
update Deneb for latest builder-specs flow (#5598)
The `BlobSidecar` construction has been moved to the relay and is no
longer done by the BN / VC in blinded flow. Builder bid contents have
been shrinked from full `BlindedBlobBundle` to `blob_kzg_commitments`.

- https://github.com/ethereum/builder-specs/pull/90
- https://github.com/ethereum/beacon-APIs/pull/369
2023-11-15 16:20:13 -08:00
Etan Kissling 0919ff05c9
reduce code duplication when collecting LC data (#5601)
Replace sections that need to be maintained with every `ConsensusFork`
related to LC data collection with a generic logic that keeps working
when unrelated parts of Ethereum change.
2023-11-15 03:21:03 +01:00
Eugene Kabanov 9889b840ce
VC: new scoring functions. (#5447)
* Initial commit.

* Fix issues and tests.

* Fix test compilation issue.

* Update AllTests.

* Change the most poor score name from <lowest> to <bad>.
Split sync committee message score in range, so lexicographic scores will not intersect with normal one.
Lexicographic scores should be below to normal scores.

* Address review comments.
Fix aggregated attestation scoring to use MAX_VALIDATORS_PER_COMMITTEE.
Fix sync committee contributions to use SYNC_SUBCOMMITTEE_SIZE.
Add getUniqueVotes test vectors.

* Post-rebase fixes.

* Address review comments.

* Return back score calculation based on actual bits length.

* AllTests modification.
2023-11-14 12:13:26 +01:00
tersec 54bdda13b4
rm unused code (#5596) 2023-11-11 11:49:34 +03:00
tersec 14c68ea285
stop calling engine_exchangeTransitionConfigurationV1 with Dencun-ready ELs (#5585) 2023-11-11 11:47:03 +03:00
tersec b669f23c25
remove `XDeclaredButNotUsed` exclusion from Nim configurations (#5595)
* remove XDeclaredButNotUsed exclusion from Nim configurations

* copyright year linting
2023-11-11 07:54:52 +00:00
tersec 1ceac20ea6
disallow htr on Opt[Foo] (#5593) 2023-11-11 05:33:54 +00:00
tersec c96163dbca
update some consensus spec URLs to v1.4.0-beta.4 (#5594) 2023-11-11 06:27:53 +01:00
Etan Kissling 9ec7b700a8
bump `gnosis-chain-configs` to `b3fc927da3e790d989edb71bec99c1db00529307` (#5589)
- Ensure all config vars are present
- Remove preset files
- ensure config is synced
2023-11-10 18:04:41 +01:00
tersec c7952ff778
bump nim-eth to remove ValidIpAddress and replace with IpAddress (#5587) 2023-11-10 16:58:48 +01:00
Etan Kissling eb35039704
allow higher `MIN_EPOCHS_FOR_BLOCK_REQUESTS` than safe minimum (#5590)
Gnosis uses `MIN_EPOCHS_FOR_BLOCK_REQUESTS` = 33024, but the computed
safe minimum (that Nimbus was using) is 2304. Relax the compatibility
check to allow `MIN_EPOCHS_FOR_BLOCK_REQUESTS` above the safe minimum
and honor `config.yaml` preferences for `MIN_EPOCHS_FOR_BLOCK_REQUESTS`.
2023-11-10 15:04:55 +00:00
Jacek Sieka f48ce6c0c7
Merge remote-tracking branch 'origin/stable' into unstable 2023-11-10 14:25:00 +01:00
Etan Kissling f388d76e49
bump `gnosis-chain-configs` to `b8ae3091439131949a994d638d730b5a5cb60f7a` (#5581)
- fix(ci): specify enr-cli version
- Update deneb config vars
2023-11-10 10:16:30 +01:00
Etan Kissling 101b723f52
make `MIN_EPOCHS_FOR_BLOB_SIDECARS_REQUESTS` configurable (#5582)
* make `MIN_EPOCHS_FOR_BLOB_SIDECARS_REQUESTS` configurable

Gnosis uses custom `MIN_EPOCHS_FOR_BLOB_SIDECARS_REQUESTS` to account
for the faster slot timing, so that blobs still remain available for
roughly the same amount of real time.

Also extend REST config endpoint with full config form `v1.4.0-beta.4`,
and extend compatibility checks when loading configs to reduce warnings.
2023-11-09 23:41:17 +03:00
tersec 68a77926e8
rm ropsten and rinkleby chain IDs from EL manager (#5586) 2023-11-09 23:35:40 +03:00
Etan Kissling d9031d97dc
rename `_INDEX` generalized index constants to `_GINDEX` (#5583)
Rename constants for consistency with upcoming spec adjustment:

- https://github.com/ethereum/consensus-specs/pull/3537
2023-11-09 12:54:44 +00:00
Eugene Kabanov c279bd4474
Fix issue when peers was not disconnected on low score. (#5579) 2023-11-08 13:42:50 +00:00
Eugene Kabanov 6bc038e8d5
VC: Obol middleware support (#5375) 2023-11-08 14:03:51 +02:00
tersec 7e3aeaea09
automated consensus spec URL updating to v1.4.0-beta.4 (#5577) 2023-11-08 05:28:03 +00:00
Eugene Kabanov 49c851109e
VC: Add pruning slashing database. (#5551)
* Add slashing database pruning to VC.
Fix GetBlockHeaderResponse object declaration (spec has been changed).

* Switch to getFinalizedBlockHeader instead.

* Fix proper sign.
Add statements.
Show pruning log statement only when pruning happens.

* Optimize and remove debugging helpers.
2023-11-06 16:40:44 +02:00
Zahary Karadjov d19ffcaa0d
Version 23.10.1 2023-11-06 16:27:05 +02:00
Etan Kissling d8a7f0df81
update Deneb for blob sidecar inclusion proofs (#5565)
`BlobSidecar` is no longer signed, instead use Merkle proof to link
blobs with block.

- https://github.com/ethereum/consensus-specs/pull/3531

Associated beacon-API / builder-specs still TBD; minimal changes done
to compile in similar style to previous spec, but not standardized yet.

- https://github.com/ethereum/beacon-APIs/pull/369
- https://github.com/ethereum/builder-specs/pull/90
2023-11-06 07:48:43 +01:00
xiaolou86 40196b75d0
docs: fix typos (#5571) 2023-11-06 03:56:07 +00:00
Etan Kissling e4dacc30c4
fix validation of blob index against gossip topic (#5569)
In `v1.4.0-alpha.0`, the blob index validation on gossip was changed to
use `compute_subnet_for_blob_sidecar` instead of having a separate topic
for each individual blob. We updated the spec reference in #5363 without
updating the code accordingly. Fixing this now, and also adding the new
`MAX_BLOBS_PER_BLOCK` check from `v1.4.0-beta.3` as it shares the theme.
2023-11-05 09:14:47 +01:00
Etan Kissling 28b84ff93b
extend `hasBlob` to all blocks from same proposer and slot (#5568)
`v1.4.0-beta.4` made the Gossip rules more strict and now requires to
ignore blobs from other branches if there are equivocating blocks.
Those blobs are only requestable via Req/Resp.
2023-11-05 09:13:57 +01:00
Etan Kissling f14389bb84
avoid perpetually sending blobs to peers (#5563)
Fix regression from #4808 where blobs that are already known are issued
ACCEPT verdict, propagating them to peers over and over again.

`validateBlobSidecar` contains the correct IGNORE logic. Moved it above
the expensive checks to retain the performance of the check.
2023-11-04 20:36:12 +01:00
Etan Kissling 87a37a32e3
introduce `BlobId` type for tracking blob subnets (#5560)
Instead of mixing up `SubnetId` (attestation subnet type) for blobs,
introduce dedicated `BlobId` type.
2023-11-04 14:20:34 +00:00
Etan Kissling a05278e263
wrap `kzgs`/`proofs`/`blobs` fields as `BlobsBundle` (#5562)
Less type conversion / copying by keeping the `BlobsBundle` together.
2023-11-04 13:49:58 +00:00
Etan Kissling fb18f09e55
align `Blobs` limit for SSZ transport (#5558)
Capacity should be set to theoretical limit to ensure correct hash root.
Actual length may be shorter. Only use is `ExecutionPayloadForSigning`
so it doesn't matter yet in practice, but still worth fixing.
2023-11-04 10:25:06 +01:00
Eugene Kabanov 8cec3af61c
VC: Obtain randao signature before slot proposal. (#5490)
* Randao calculation caching for VC implementation.

* Add time monitoring for randao signatures process.

* Add delay calculation.

* Address review comments.

* Address review comments.
2023-11-04 08:14:14 +01:00
Etan Kissling 29fe958908
allow to pass trusted node sync options during BN startup (#5545)
Using trusted node sync currently requires to run two commands -
first the `trustedNodeSync` command to initialize the database,
followed by the regular startup command to continue syncing.

The `trustedNodeSync` options are now also available during regular
startup, and are used when the database is empty to simplify setting up
a new Nimbus beacon node. This also aligns behaviour closer with other
Ethereum consensus implementations.

The new logic only applies if the database has not yet been initialized;
same as before. Also, the database needs to be removed under the same
conditions as before when a fresh sync is desired.
2023-11-03 16:07:49 +01:00
Etan Kissling c95f9feec4
remove unused Deneb code (#5556)
The `process_blob_kzg_commitments` step was removed in `v1.4.0-alpha.1`,
but we haven't deleted the now unused functions. Do that now.
2023-11-03 14:37:44 +00:00
Etan Kissling baf81595a7
add `KZG_COMMITMENT_INCLUSION_PROOF_DEPTH` preset constant (#5555)
`v1.4.0-beta.4` added `KZG_COMMITMENT_INCLUSION_PROOF_DEPTH` to preset:

- Spec PR: https://github.com/ethereum/consensus-specs/pull/3531
- Gnosis PR: https://github.com/gnosischain/configs/pull/17
2023-11-03 13:54:21 +00:00
tersec ab89e96265
fill in transactions root and KZG commitments from deneb.BlindedBeaconBlock in state transition (#5553) 2023-11-02 04:56:04 +01:00
tersec 657de2721a
fill blindedblockbody KZG commitments from deneb blinded header (#5550) 2023-11-01 14:00:21 +01:00
tersec ed446b1d5a
replace ValidIpAddress with IpAddress in configurations (#5536) 2023-11-01 08:33:00 +01:00
Eugene Kabanov e1f47bb27a
Address #5539. (#5548) 2023-11-01 08:32:41 +01:00
Eugene Kabanov 1a5bcb479e
Fix `broadcast_validation` handling in `publishBlockV2` (fixes #5531) (#5547)
* Address issue #5531.

* Add more tests.

* Add to resttest ability to check values.
Fix tests.
2023-11-01 08:31:18 +01:00
Eugene Kabanov 70aaeee704
Enable comprehensive reasons when REST server faults to create server. (#5546)
Bump chronos & presto.
2023-11-01 08:27:43 +01:00
tersec 556d5e7114
rm unused code (#5538) 2023-11-01 05:53:09 +01:00
Etan Kissling 173582ad70
initialize KZG crypto before starting network (#5543)
Move KZG trusted setup initialization before `BeaconNode.init` to avoid
edge case where network message is received and processed before crypto
library has been properly initialized. Followup from #4870.
2023-10-31 13:43:46 +01:00
Jacek Sieka 8c81515bf6
log slashing protection database error on failure to load (#5542)
* log slashing protection database error on failure to load

...and fix noreturn warning

* fix the copied one too

* oops

* oops 2
2023-10-31 11:15:38 +01:00
Etan Kissling d289da8cd4
extract trusted node sync trigger into separate function (#5544)
To prepare for calling trusted node sync from the main Nimbus startup
logic, extract the trusted node sync trigger into a separate function.
Further allow passing a pre-opened database, as that will be needed to
check whether trusted node sync needs to be called during regular start.
2023-10-31 01:56:52 +01:00
tersec 62d59daaa7
consensus-spec URL updates to v1.4.0-beta.3 (#5541) 2023-10-30 06:44:43 +00:00
tersec b0d5001571
implement getBlobSidecars Beacon API endpoint (#5530) 2023-10-29 00:06:13 +00:00
tersec 34eade3a99
sign blinded blob sidecars during builder API proposals (#5537) 2023-10-28 05:10:23 +00:00
tersec 922283e2e3
clean up ncli_testnet and reduce `ValidIpAddress` usage (#5529)
* ValidIpAddress -> IpAddress

* clean up ncli_testnet and reduce ValidIpAddress usage
2023-10-26 23:16:15 +00:00
tersec 09df3f32b5
add non-SZ getBlobSidecar and BlobSidecar database tests (#5528) 2023-10-26 03:40:04 +00:00
tersec 41dfc9ed91
add `KZGProof` and `Blob` REST JSON serializations (#5526)
* add KZGProof and Blob REST JSON serializations

* error summary updates

* copyright notice

* consistent integer types for Nim 2.0
2023-10-25 23:50:59 +02:00
Etan Kissling d32fafa2ec
`investigated` -> `inspected` in `libnimbus_lc` docs (#5523)
Consistently use `inspected` instead of `investigted` terminology in
documentation of `libnimbus_lc` functions.
2023-10-24 16:09:44 +00:00
tersec 5aa5095e17
Fix REST JSON KzgCommitment encoding/decoding to use hex strings per spec (#5522) 2023-10-24 15:22:59 +00:00
Etan Kissling 468ee6a7ed
simplify `getBlockSSZ` and `getBlockSZ` implementation (#5521)
Use `forks` sugar to reduce code duplication in `beacon_chain_db.nim`.
2023-10-24 09:15:34 +00:00
tersec 513c9d9637
fix Deneb RestPublishedSignedBlockContents REST JSON decoding (#5520) 2023-10-24 05:50:32 +00:00
tersec 026ad21906
send DenebBlockContents in produceBlockV2 (#5519) 2023-10-24 02:50:26 +00:00
tersec fd21882e24
improve test coverage of post-Bellatrix initialize_beacon_state_from_eth1 (#5517) 2023-10-24 01:58:52 +00:00
tersec b2e81de6db
remove unnecessary Builder API status endpoint usage (#5518) 2023-10-23 23:01:29 +00:00
tersec a986a1bcd8
using deneb_mev for Deneb is less incorrect than using capella_mev (#5516) 2023-10-20 01:39:47 +00:00
tersec 769316d909
construct Deneb blinded blobs from builder API getHeader (#5513) 2023-10-19 17:34:32 +00:00
tersec 4ddd771127
automated consensus spec URL updating to v1.4.0-beta.3 (#5514) 2023-10-19 10:26:38 +00:00
tersec 9fd52ebe9f
use consensus spec v1.4.0-beta.3 test vectors (#5512) 2023-10-19 05:14:02 +00:00
Zahary Karadjov d52e8165d5
Merge branch 'stable' into unstable 2023-10-18 20:10:00 +03:00
tersec 6eb7d481fa
fix Deneb builder API block body type; introduce and use ExecutionPayloadHeaderAndBlindedBlobsBundle (#5509) 2023-10-18 02:54:02 +00:00
andri lim 0f9b52933e
Bump nim-eth: Change block timestamp from std.Time to distinct uint64 (#5510)
* Bump nim-eth: Change block timestamp from std.Time to distinct uint64

Also change tx.maxFeePerBlobGas from GasInt to UInt256
following Cancun latest spec

* Fix EthTime.now from func to proc due to sideeffects
2023-10-18 07:37:57 +07:00