Commit Graph

1625 Commits

Author SHA1 Message Date
Etan Kissling 87a37a32e3
introduce `BlobId` type for tracking blob subnets (#5560)
Instead of mixing up `SubnetId` (attestation subnet type) for blobs,
introduce dedicated `BlobId` type.
2023-11-04 14:20:34 +00:00
Etan Kissling a05278e263
wrap `kzgs`/`proofs`/`blobs` fields as `BlobsBundle` (#5562)
Less type conversion / copying by keeping the `BlobsBundle` together.
2023-11-04 13:49:58 +00:00
Etan Kissling fb18f09e55
align `Blobs` limit for SSZ transport (#5558)
Capacity should be set to theoretical limit to ensure correct hash root.
Actual length may be shorter. Only use is `ExecutionPayloadForSigning`
so it doesn't matter yet in practice, but still worth fixing.
2023-11-04 10:25:06 +01:00
Etan Kissling c95f9feec4
remove unused Deneb code (#5556)
The `process_blob_kzg_commitments` step was removed in `v1.4.0-alpha.1`,
but we haven't deleted the now unused functions. Do that now.
2023-11-03 14:37:44 +00:00
Etan Kissling baf81595a7
add `KZG_COMMITMENT_INCLUSION_PROOF_DEPTH` preset constant (#5555)
`v1.4.0-beta.4` added `KZG_COMMITMENT_INCLUSION_PROOF_DEPTH` to preset:

- Spec PR: https://github.com/ethereum/consensus-specs/pull/3531
- Gnosis PR: https://github.com/gnosischain/configs/pull/17
2023-11-03 13:54:21 +00:00
tersec ab89e96265
fill in transactions root and KZG commitments from deneb.BlindedBeaconBlock in state transition (#5553) 2023-11-02 04:56:04 +01:00
tersec 657de2721a
fill blindedblockbody KZG commitments from deneb blinded header (#5550) 2023-11-01 14:00:21 +01:00
Eugene Kabanov e1f47bb27a
Address #5539. (#5548) 2023-11-01 08:32:41 +01:00
Eugene Kabanov 1a5bcb479e
Fix `broadcast_validation` handling in `publishBlockV2` (fixes #5531) (#5547)
* Address issue #5531.

* Add more tests.

* Add to resttest ability to check values.
Fix tests.
2023-11-01 08:31:18 +01:00
tersec 556d5e7114
rm unused code (#5538) 2023-11-01 05:53:09 +01:00
tersec 62d59daaa7
consensus-spec URL updates to v1.4.0-beta.3 (#5541) 2023-10-30 06:44:43 +00:00
tersec 34eade3a99
sign blinded blob sidecars during builder API proposals (#5537) 2023-10-28 05:10:23 +00:00
tersec 41dfc9ed91
add `KZGProof` and `Blob` REST JSON serializations (#5526)
* add KZGProof and Blob REST JSON serializations

* error summary updates

* copyright notice

* consistent integer types for Nim 2.0
2023-10-25 23:50:59 +02:00
tersec 5aa5095e17
Fix REST JSON KzgCommitment encoding/decoding to use hex strings per spec (#5522) 2023-10-24 15:22:59 +00:00
tersec 513c9d9637
fix Deneb RestPublishedSignedBlockContents REST JSON decoding (#5520) 2023-10-24 05:50:32 +00:00
tersec fd21882e24
improve test coverage of post-Bellatrix initialize_beacon_state_from_eth1 (#5517) 2023-10-24 01:58:52 +00:00
tersec b2e81de6db
remove unnecessary Builder API status endpoint usage (#5518) 2023-10-23 23:01:29 +00:00
tersec a986a1bcd8
using deneb_mev for Deneb is less incorrect than using capella_mev (#5516) 2023-10-20 01:39:47 +00:00
tersec 4ddd771127
automated consensus spec URL updating to v1.4.0-beta.3 (#5514) 2023-10-19 10:26:38 +00:00
tersec 9fd52ebe9f
use consensus spec v1.4.0-beta.3 test vectors (#5512) 2023-10-19 05:14:02 +00:00
tersec 6eb7d481fa
fix Deneb builder API block body type; introduce and use ExecutionPayloadHeaderAndBlindedBlobsBundle (#5509) 2023-10-18 02:54:02 +00:00
andri lim 0f9b52933e
Bump nim-eth: Change block timestamp from std.Time to distinct uint64 (#5510)
* Bump nim-eth: Change block timestamp from std.Time to distinct uint64

Also change tx.maxFeePerBlobGas from GasInt to UInt256
following Cancun latest spec

* Fix EthTime.now from func to proc due to sideeffects
2023-10-18 07:37:57 +07:00
zah 35bf03a3fb
Add the --verifying-web3-signer-url configuration option (#5504) 2023-10-13 15:42:00 +03:00
tersec eeac00d08e
add RPC signature; add type constraints on proposeBlockMEV (#5498) 2023-10-12 11:49:48 +00:00
tersec 40616b6243
update some consensus spec URLs to v1.4.0-beta.2 (#5497) 2023-10-12 05:07:41 +00:00
tersec edc5c03230
consistently use SignedBlindedBeaconBlockContents; remove more Bellatrix Builder API remnants (#5493) 2023-10-11 11:48:35 +00:00
tersec 48197e4d55
add/update Deneb Builder API types and RPC signatures (#5492) 2023-10-11 04:25:59 +00:00
tersec 447786518f
ShufflingRef approach to next-epoch validator duty calculation/prediction (#5414)
* ShufflingRef approach to next-epoch validator duty calculation/prediction

* refactor action_tracker.updateActions to take ShufflingRef + beacon_proposers; refactor maybeUpdateActionTrackerNextEpoch to be separate and reused function; add actual fallback logic

* document one possible set of conditions

* check epoch participation flags and inactivity scores to ensure no penalties and MAX_EFFECTIVE_BALANCE to ensure rewards don't matter

* correctly (un)shuffle each proposer index

* remove debugging assertion
2023-10-10 00:02:07 +00:00
Etan Kissling dd1f362536
align `Beacon(Block(Body)?|State)Type` with other fork sugar (#5483)
The templates for `BeaconBlock`, `BeaconBlockBody` and `BeaconState`
are the only ones using a `macro` mechanism for code generation.
This prevents using the dot-syntax style `consensusFork.BeaconFoo`
in some situations, and also tends to trigger naming conflicts,
requiring the `Type` suffix. Furthermore, the `macro` only works
for types that are re-defined in every single `ConsensusFork`.

Replacing with the simpler but more verbose approach used for other
types for consistency and to avoid the downsides of the `macro`.

Furthermore, simplify `test_fixture_sanity_blocks` to use `forks` sugar.
2023-10-05 14:01:40 +02:00
Etan Kissling 297c768816
reduce redundant zero initialization for LC data objects (#5479)
Directly initialize `ForkedLightClientObj` instead of separately first
 setting the `kind` (initializing everything to zero) and then assigning
the forky data after that.
2023-10-04 18:11:45 +02:00
Etan Kissling 8aaea1a14e
simplify fork choice test (#5467)
Reduce overly verbose fork branching logic in fork choice test with
generic implementation based on `consensusFork`.
2023-09-29 13:26:34 +00:00
Jacek Sieka 252473d2d1
ssz: bump (#5463)
more `zeroMem` avoidance, value semantics for merkleizer
2023-09-28 11:59:33 +00:00
Etan Kissling 7c45b8f98d
convert remaining `toFork` -> `kind` for consistency (#5462)
We currently have multiple ways to obtain `ConsensusFork` or
`LcDataFork` from a forky type. Rename `toFork` to `kind`
for a consistent API naming.
2023-09-27 15:10:28 +00:00
Eugene Kabanov 4fb95d000d
REST server fixes and improvements. (#5422)
* Move from Option[T] to Opt[T] usage.

* Add `finalized` flag.

* Fix compilation issue.

* Http415 error code for some REST API calls.
Introduce more comprehensive error reporting for block calls.
Deprecate decodeEthConsensusVersion() function.

* Bump http-utils.

* Fix copyright year.

* Fix serialization issue.

* Address review comments.

* Post rebase fixes.
2023-09-27 16:45:33 +02:00
tersec 710f26745d
small attestation pool clean up (#5458) 2023-09-24 08:50:48 +00:00
Eugene Kabanov e3fe762ec9
Fix unknown fields should be ignored when parsing liveness responses. (#5418) 2023-09-23 08:44:54 +02:00
Etan Kissling 47639ef89c
replace `Eth2NetworkMetadata` case object with `Result` (#5452)
`Eth2NetworkMetadata` has an `incompatible` case to hold an error string
in case the loaded file is not compatible with the compile-time config.
The same can be modeled with a `Result[Eth2NetworkMetadata, string]` and
avoids followup checks for the `incompatible` case.
2023-09-22 23:45:24 +02:00
Etan Kissling 5150505e66
bump `nim-ssz-serialization` to `a5767c1b4992dca3e5bd72557aadd4df0f183bf8` (#5457)
- nimble/ci updates
- use common super-type in `raisesssz` pragma
2023-09-22 23:44:57 +02:00
tersec 2895a9a05c
automated consensus spec URL updating to v1.4.0-beta.2 (#5453) 2023-09-21 18:06:51 +00:00
Etan Kissling e7bc41e005
`blck` --> `forkyBlck` when using `withBlck` / `withStateAndBlck` (#5451)
For symmetry with `forkyState` when using `withState`, and to avoid
problems with shadowing of `blck` when using `withBlck` in `template`,
also rename the injected `blck` to `forkyBlck`.

- https://github.com/nim-lang/Nim/issues/22698
2023-09-21 12:49:14 +02:00
Etan Kissling 273f1d34cd
implement EIP-7514 for Deneb: Add Max Epoch Churn Limit (#5442)
* implement EIP-7514 for Deneb: Add Max Epoch Churn Limit

Cap activations per epoch according to EIP-7514:

- https://eips.ethereum.org/EIPS/eip-7514
- https://github.com/ethereum/consensus-specs/pull/3499

* apply proposer boost to first block in case of equivocation

Implement spec changes to fork choice; this only affects equivocation
when multiple blocks are signed for the same slot. Regular operation
is not changed.

- https://github.com/ethereum/consensus-specs/pull/3352

* bump test vectors to v1.4.0-beta.2-hotfix

---------

Co-authored-by: tersec <tersec@users.noreply.github.com>
2023-09-20 13:39:56 +02:00
Etan Kissling 1640c45a95
reject invalid proofs when validating blobs (#5445)
Currently, passing `0xc00000...` proof seems to pass `verifyProofs`.
Unsure why such a check is not necessary in spec, and also unsure
whether it is correct to reject proof at infinity, or if it could
occur, e.g., for a blob containing all 0 bytes. Weird overall...

* proper fix
2023-09-19 22:13:58 +00:00
tersec ee75d45a8b
fix `ATTESTATION_PROPAGATION_SLOT_RANGE` unknown constant warning (#5433)
* fix ATTESTATION_PROPAGATION_SLOT_RANGE unknown constant warning

* verify underflow not an issue
2023-09-19 06:54:58 +00:00
tersec 5a29ad7e4f
update some consensus-spec URLs to v1.4.0-beta.1 (#5429) 2023-09-14 18:23:59 +00:00
tersec 7a001d4896
remove 4 more missing constants in preset handling warning (#5428) 2023-09-14 12:37:35 +00:00
andri lim 5c88e74c08
Bump stint to v2.0: new array backend (#5113)
* bump stint to v2.0: new array backend

* Fix missing isEven in libnimbus_lc.nim

* bump nim-stint: compiles with arm64
2023-09-13 01:16:04 +07:00
Etan Kissling 8fa5580c50
add infrastructure to select fork choice version (#5387)
To allow testing https://github.com/ethereum/consensus-specs/issues/3466
add support for selecting fork choice version at launch. This means we
can deploy a different logic when `DENEB_FORK_EPOCH != FAR_FUTURE_EPOCH`
that won't be used on Mainnet.
2023-09-12 09:52:51 +02:00
tersec 2b4f987c80
remove pre-v1.4.0 attestation stability subnets (#5402)
* remove pre-v1.4.0 attestation stability subnets

* re-add most of should register stability subnets on attester duties test
2023-09-11 16:03:34 +00:00
Eugene Kabanov 3c3c4e8edf
Fix /eth/v1/validator/liveness/{epoch} call names and links in code (#5392)
* Fix getLiveness() call links and names.

* Add getLiveness() decoding test.

* Remove BN state updates from getLiveness() call.

* Fix AllTests.

* Reduce number of tests.
2023-09-11 12:07:34 +02:00
tersec 10ec7be686
optimize epoch registry processing (#5412) 2023-09-11 11:21:50 +02:00
tersec cc13e0b7e0
restore full test coverage for process_rewards_and_penalties (#5407)
* restore full test coverage for process_rewards_and_penalties

* adjust ncli_db to use new iterator
2023-09-08 16:42:18 +00:00
zah 53589b5a7d
Add metadata for the Holesky network (#5337)
* Add metadata for the Holesky network

* Add copyright banner to the new Nim module

* Working version

* Bump Chronos to fix downloading from Github
* Add checksum check of the downloaded file
* Clean up debugging code and obsolete imports
2023-09-08 08:53:27 +03:00
tersec f53422c181
optimize epoch transition via get_flag_index_deltas() and get_inactivity_penalty_deltas() (#5404)
* optimize epoch transition via get_flag_index_deltas() and get_inactivity_penalty_deltas()

* directly mutate Altair info in {get,update}_flag_and_inactivity_deltas
2023-09-07 11:14:52 +00:00
Etan Kissling 89d133d0e5
further reduce stack size of LC helpers for Nim 2.0 (#5401)
The `is_next_sync_committee_known` helper allocates a fresh
`SyncCommittee` in the caller and `nimZeroMem`s it on each use.
Use `static` to compare against a compile-time zeroed copy instead.
This also should help reduce stack size far enough to link with Nim 2.0.
2023-09-06 16:05:12 +02:00
Eugene Kabanov 4ce8e77b56
Address issue #5362. (#5370)
* Address issue #5362.

* Remove RestSpec object declaration and getSpec() API call declaration.

* Address review comments.
2023-09-05 23:39:01 +03:00
Etan Kissling dbb1a63ca9
collapse `partialBeaconBlock` templates into one (#5386)
Have a single `Forky` template for `partialBeaconBlock` production
rather than have several copies that are mostly identical.
2023-09-05 15:59:17 +02:00
Etan Kissling be9ecfa1c9
add helpers for processing withdrawals to `libnimbus_lc.a` (#5374)
Similar to the existing helpers for processing transactions / receipts,
extend `libnimbus_lc.a` with support for processing withdrawals as well.
2023-09-04 20:44:03 +02:00
tersec a8c56b1660
update some consensus-spec URLs to v1.4.0-beta.1 (#5379) 2023-09-01 09:31:52 +00:00
Eugene Kabanov 757328372a
Dynamic validators set. (#5366)
* Initial commit.

* Fix argument to be optional.

* Adopt options.md.
2023-08-31 15:16:15 +03:00
tersec 6c42cfa79e
use correct decimal or hex encoding/decoding for REST GetSpec (#5371) 2023-08-30 12:43:25 +03:00
Etan Kissling 09020ebd2f
add helpers for processing receipts to `libnimbus_lc.a` (#5360)
Similar to the existing helpers for processing transactions,
extend `libnimbus_lc.a` with support for processing receipts as well.
2023-08-28 15:56:40 +00:00
tersec db6f4e8090
update some consensus-spec URLs to v1.4.0-beta.1 (#5357) 2023-08-25 15:58:44 +00:00
Etan Kissling 8ff0f623b4
change unused template to unused func (#5355)
The intentionally unused `toSszType` template may trigger warnings
as it is unused. Replacing it with an `{.error.}` `func` prevents that.
2023-08-25 11:09:13 +00:00
Etan Kissling d8f44ed905
fix LC header validation in Deneb (#5350)
`blob_gas_used` must be 0 before Deneb.
2023-08-25 10:36:10 +00:00
Etan Kissling 991c31f42b
add helpers for processing transactions to `libnimbus_lc.a` (#5269)
It is useful to verify transactions data against `transactionsRoot`.
Add corresponding functionality to the light client library.
2023-08-25 11:29:39 +02:00
Etan Kissling c211a3849e
remove `{.raises: [Defect].}` Nim 1.2 compatibility (#5352)
In Nim 1.6, `{.raises: [Defect].}` is no longer necessary. Remove it.
2023-08-25 11:29:07 +02:00
Etan Kissling 9637d3f8d0
remove unused `type` in `makeLimitedUInt` (#5354)
The `type L` definition inside `makeLimitedUInt` is unused. Remove it.
2023-08-25 11:28:52 +02:00
Etan Kissling 758c503a52
move mock genesis block hash to `tests` (#5353)
From old interop tests, a mock `eth1BlockHash` was defined in `base`.
To avoid accidental use by Nimbus, move to `tests` and rename it to
`mockEth1BlockHash`.
2023-08-25 11:28:42 +02:00
tersec ef184c8959
subscribe to Deneb blob subnets (#5342) 2023-08-23 19:31:41 +00:00
Jacek Sieka a5151bc546
`validator_duties`->`beacon_validators` (#5333)
This PR renames the existing `validator_duties` to `beacon_validators`
and in doing so, names validators running inside the beacon node process
"beacon validators" while those running the VC can be referred to as
"client validators" to disambiguate the two.

The existing `validator_duties` instead takes on a new responsibility:
as a home for logic shared between beacon and client validators - ie
code that provides consistency in implementation and behavior between
the two modes of operation.

Not only does this simplify reasoning about where to put code -it also
reduces the number of dependencies the validator client has from ~5000
to ~3000 modules (!) according to `nim genDepend` significantly reducing
compile times.
2023-08-23 19:39:57 +03:00
tersec ec30b0cc77
rm Bellatrix builder API support remnant (#5340) 2023-08-23 19:35:48 +03:00
tersec c08d125fe3
treat more constants as quasi-runtime-presets per v1.4.0 consensus specs (#5335) 2023-08-21 19:58:51 +00:00
Etan Kissling 2dae8ea26e
cleanup `{.raises.}` for `writeValue` (#5308)
- Remove unnecessary `Defect` references
- Remove spurious `SerializationError` references
- Remove duplicate `writeValue` template in `keystore.nim`;
  same implementation already exists a bit further above in same file.
2023-08-19 17:11:56 +02:00
Etan Kissling 8e500aa62f
allow filtering fee recipient logs by topic (#5320)
Add separate log topic for `Refreshing fee recipient` messages.
2023-08-19 11:16:30 +00:00
tersec 6fd4983f6b
EIP-4788/parentBeaconBlockRoot fixes (#5318) 2023-08-19 08:38:17 +00:00
tersec 0d1614326d
16-byte align ValidatorSig and TrustedSig (#5301) 2023-08-17 06:27:26 +00:00
tersec 22b7b721d0
adjust digest hashing not to care about alignment; increase ValidatorPubKey alignment (#5296) 2023-08-17 03:06:21 +00:00
tersec c3584a9ea7
fix regression in remote signing REST schema (#5294) 2023-08-15 22:39:12 +00:00
tersec d171303133
update some consensus spec URLs to v1.4.0-beta.1 (#5287) 2023-08-12 10:38:06 +00:00
tersec 6cd8f3ba55
Update constants/presets for consistency with consensus-specs v1.4.0 (#5284) 2023-08-12 03:10:12 +00:00
henridf 9efd26c2e9
Add support for POST /eth/v2/beacon/blocks (#5214)
* Add support for POST /eth/v2/beacon/blocks

* More descriptive errors

* Address review feedback

* Return 500 (not 400) for a missing implementation case
2023-08-12 03:08:54 +00:00
Etan Kissling 973e9e9a98
speed up LC signature verification (#5268)
We know the aggregate publickey of a fully participating sync committee.
Because participation is typically very high (>95%), it is faster to
start from that aggregate publickey and subtract the individual keys of
non-participants, than summing up all the participating pubkeys.
2023-08-09 08:50:07 +02:00
tersec 85e1976ac3
automated consensus spec URL updating to v1.4.0-beta.1 (#5280) 2023-08-09 03:58:47 +00:00
tersec 8c3d1cd3aa
ensure `ValidatorPubKey` int-compatibly aligned to fix UB (#5277)
* ensure ValidatorPubKey int-compatibly aligned

* use sizeof(Hash) explicitly
2023-08-08 19:01:32 +00:00
tersec 18061f7991
Remove most remaining portions of Bellatrix Builder API (#5272) 2023-08-08 00:05:17 +00:00
Etan Kissling 6a2bac5cee
need even more log lines for debugging keymanager (#5260)
Keymanager test logs are still cut off, further increase log lines.
2023-08-07 18:02:08 +02:00
Etan Kissling d7afa1c78a
add helpers for processing EL block header to `libnimbus_lc.a` (#5199)
To obtain the correct `transactions_root` and `withdrawals_root`,
it is necessary to process execution block header. Light client updates
don't contain the correct MPT roots.
2023-08-07 14:23:44 +02:00
tersec 2c424cac87
Merkle is someone's name (#5265) 2023-08-07 10:06:47 +00:00
tersec 2ae2d0a042
remove Bellatrix Builder API (Signed)BuilderBid (#5262)
* remove Bellatrix Builder API (Signed)BuilderBid

* stub out BlindedBeaconBlock
2023-08-06 09:20:43 +00:00
tersec 1f5dd83fbd
unplumbing Bellatrix builder API support (#5203) 2023-08-05 05:13:08 +00:00
tersec d542410a17
remove more Bellatrix-only Builder API support infrastructure (#5251) 2023-08-05 00:11:51 +00:00
Jacek Sieka b8a32419b8
async batch verification (+40% sig verification throughput) (#5176)
* async batch verification

When batch verification is done, the main thread is blocked reducing
concurrency.

With this PR, the new thread signalling primitive in chronos is used to
offload the full batch verification process to a separate thread
allowing the main threads to continue async operations while the other
threads verify signatures.

Similar to previous behavior, the number of ongoing batch verifications
is capped to prevent runaway resource usage.

In addition to the asynchronous processing, 3 addition changes help
drive throughput:

* A loop is used for batch accumulation: this prevents a stampede of
small batches in eager mode where both the eager and the scheduled batch
runner would pick batches off the queue, prematurely picking "fresh"
batches off the queue
* An additional small wait is introduced for small batches - this helps
create slightly larger batches which make better used of the increased
concurrency
* Up to 2 batches are scheduled to the threadpool during high pressure,
reducing startup latency for the threads

Together, these changes increase attestation verification throughput
under load up to 30%.

* fixup

* Update submodules

* fix blst build issues (and a PIC warning)

* bump

---------

Co-authored-by: Zahary Karadjov <zahary@gmail.com>
2023-08-03 11:36:45 +03:00
Jacek Sieka 5bc48acc36
reduce memory allocations during state transition (#5235)
This PR removes a few hundred thousand temporary seq allocations during
state transition - in particular, the flag seq was allocated per
validator while committees are computed per attestation.
2023-08-03 01:03:40 +02:00
henridf 28194468c9
Rename "data gas" to "blob gas" (#5216)
* Rename data gas to blob gas

* Update vendor/nim-eth and vendor/nim-web3
2023-08-02 22:07:57 +00:00
tersec 909a7c2d07
use v1.4.0-beta.1 consensus spec tests (#5230) 2023-07-31 21:51:06 +00:00
Etan Kissling 971b4483c5
add `libnimbus_lc.a` C library (#5122)
Add a new C library for processing light client data based on the Nimbus
implementation. This can be used from other, non-Nimbus components.
2023-07-19 09:48:39 +02:00
Etan Kissling dd35d2d309
delay `onSlotEnd` if there are duties (#5196)
We currently call `onSlotEnd` whenever all in-BN validator duties are
completed. VC validator duties are not awaited. When `onSlotEnd` is
processed close to the slot start, a VC may therefore miss duties.
Adding a delay before `onSlotEnd` improves this situation.
The logic can be optimized further if `ActionTracker` would track
`knownValidators` from REST separately from in-process ones.
2023-07-18 20:55:36 +02:00
tersec 565edfa351
remove unused code (#5158)
* remove unused code

* add copyright header
2023-07-15 16:30:52 +00:00
tersec 703eead2e6
ensure the arrays in BeaconStateDiff are int-indexable in Nim 2.0 (#5189) 2023-07-13 23:25:10 +02:00
tersec a1ddd447bc
use correct import name for kzg4844 (#5179) 2023-07-10 23:55:40 +00:00