Commit Graph

1840 Commits

Author SHA1 Message Date
Etan Kissling 6a2bac5cee
need even more log lines for debugging keymanager (#5260)
Keymanager test logs are still cut off, further increase log lines.
2023-08-07 18:02:08 +02:00
Etan Kissling d7afa1c78a
add helpers for processing EL block header to `libnimbus_lc.a` (#5199)
To obtain the correct `transactions_root` and `withdrawals_root`,
it is necessary to process execution block header. Light client updates
don't contain the correct MPT roots.
2023-08-07 14:23:44 +02:00
tersec 2c424cac87
Merkle is someone's name (#5265) 2023-08-07 10:06:47 +00:00
tersec 2ae2d0a042
remove Bellatrix Builder API (Signed)BuilderBid (#5262)
* remove Bellatrix Builder API (Signed)BuilderBid

* stub out BlindedBeaconBlock
2023-08-06 09:20:43 +00:00
tersec 1f5dd83fbd
unplumbing Bellatrix builder API support (#5203) 2023-08-05 05:13:08 +00:00
tersec d542410a17
remove more Bellatrix-only Builder API support infrastructure (#5251) 2023-08-05 00:11:51 +00:00
Jacek Sieka b8a32419b8
async batch verification (+40% sig verification throughput) (#5176)
* async batch verification

When batch verification is done, the main thread is blocked reducing
concurrency.

With this PR, the new thread signalling primitive in chronos is used to
offload the full batch verification process to a separate thread
allowing the main threads to continue async operations while the other
threads verify signatures.

Similar to previous behavior, the number of ongoing batch verifications
is capped to prevent runaway resource usage.

In addition to the asynchronous processing, 3 addition changes help
drive throughput:

* A loop is used for batch accumulation: this prevents a stampede of
small batches in eager mode where both the eager and the scheduled batch
runner would pick batches off the queue, prematurely picking "fresh"
batches off the queue
* An additional small wait is introduced for small batches - this helps
create slightly larger batches which make better used of the increased
concurrency
* Up to 2 batches are scheduled to the threadpool during high pressure,
reducing startup latency for the threads

Together, these changes increase attestation verification throughput
under load up to 30%.

* fixup

* Update submodules

* fix blst build issues (and a PIC warning)

* bump

---------

Co-authored-by: Zahary Karadjov <zahary@gmail.com>
2023-08-03 11:36:45 +03:00
Jacek Sieka 5bc48acc36
reduce memory allocations during state transition (#5235)
This PR removes a few hundred thousand temporary seq allocations during
state transition - in particular, the flag seq was allocated per
validator while committees are computed per attestation.
2023-08-03 01:03:40 +02:00
henridf 28194468c9
Rename "data gas" to "blob gas" (#5216)
* Rename data gas to blob gas

* Update vendor/nim-eth and vendor/nim-web3
2023-08-02 22:07:57 +00:00
tersec 909a7c2d07
use v1.4.0-beta.1 consensus spec tests (#5230) 2023-07-31 21:51:06 +00:00
Etan Kissling 971b4483c5
add `libnimbus_lc.a` C library (#5122)
Add a new C library for processing light client data based on the Nimbus
implementation. This can be used from other, non-Nimbus components.
2023-07-19 09:48:39 +02:00
Etan Kissling dd35d2d309
delay `onSlotEnd` if there are duties (#5196)
We currently call `onSlotEnd` whenever all in-BN validator duties are
completed. VC validator duties are not awaited. When `onSlotEnd` is
processed close to the slot start, a VC may therefore miss duties.
Adding a delay before `onSlotEnd` improves this situation.
The logic can be optimized further if `ActionTracker` would track
`knownValidators` from REST separately from in-process ones.
2023-07-18 20:55:36 +02:00
tersec 565edfa351
remove unused code (#5158)
* remove unused code

* add copyright header
2023-07-15 16:30:52 +00:00
tersec 703eead2e6
ensure the arrays in BeaconStateDiff are int-indexable in Nim 2.0 (#5189) 2023-07-13 23:25:10 +02:00
tersec a1ddd447bc
use correct import name for kzg4844 (#5179) 2023-07-10 23:55:40 +00:00
henridf 23edc4f960
Finish readValue for deneb RestPublishedBeaconBlockBody (#5172)
(Stray piece that got left behind a previous PR...)
2023-07-10 14:57:11 +00:00
tersec 1e2d3b12c4
update some consensus-specs ref URLs to v1.4.0-beta.0 (#5163) 2023-07-05 16:02:55 +00:00
tersec 583eb4db17
work around some CastSizes warnings and fix unreachable code warnings (#5149) 2023-06-30 21:34:04 +02:00
tersec 0be95571a7
update some consensus spec URLs to v1.4.0-beta.0 (#5150) 2023-06-30 16:12:54 +00:00
henridf 99264d7507
Local validator proposals (#5137)
* Implement local validator block proposals

* Address review feedback

* Fix rebase issue
2023-06-30 07:39:41 +00:00
tersec ba94dc849f
opt-out support for v1.4.0 attestation stability subnets (#5128)
* opt-out support for v1.4.0 attestation stability subnets

* document debug-prefix-flag-policy and remove temporary from flag name
2023-06-30 05:23:15 +00:00
tersec d5a6d51ede
Web3Signer support only for only Bellatrix and later (#5107)
Co-authored-by: Etan Kissling <etan@status.im>
2023-06-29 09:36:43 +00:00
henridf 7b0e3b1c04
Review feedbac from #5106 (#5141) 2023-06-29 01:17:54 +00:00
henridf 1234900065
Add blob handling to message router (#5106)
* Add blob handling to message router

* address review feedback

* Fix typos
2023-06-28 17:55:31 +00:00
Eugene Kabanov 5e1a0eac85
VC: Runtime config (#5130)
* Make VC able to understand any type of `/eth/v1/config/spec` response without any changes in source code.
Update compatibility checking.
Now VC is able to obtain any constant from `spec` call.

* Remove RestSpecVC declaration.
2023-06-28 15:33:38 +02:00
Eugene Kabanov c2c5d80a4f
VC: Remote BN clock offset monitoring. (#4846)
* Initial commit.

* Add algorithm in comment.
Remove delays.
Fix logging statement issues.
Change update from epoch to slot.

* Obtain timestamp earlier.

* Add processing delays into algorithm.

* Fix time offset logging to produce integers instead of strings.

* Address review comments.

* Fix copyright year.
Fix updateStatus().

* Remove fields from Slot start log statement.
Fix issues when BN do not support Nimbus Extensions.
Rename metric name and type change.

* Add beacon role to disable time offset check manually.
2023-06-28 15:33:07 +02:00
henridf ac3d8939a8
Add Deneb handling to state_transition.makeBeaconBlock() (#5118)
* Add Deneb handling to state_transition.makeBeaconBlock()

* review feedback
2023-06-26 16:05:02 +00:00
tersec a2bdc2d723
revert 2023-06-26 08:38:05 +00:00
tersec dbd7611983
not a real change 2023-06-26 08:36:39 +00:00
tersec 614202e30d
automated consensus spec URL updating to v1.4.0-beta.0 (#5121) 2023-06-24 15:43:30 +00:00
tersec 9dbf63d8ca
use v1.4.0-beta.0 consensus spec test vectors (#5120) 2023-06-24 12:27:15 +00:00
tersec b62664915e
consensus spec v1.4.0 attestation stability subnets (#5092) 2023-06-23 09:30:46 +00:00
henridf fa212515f5
Implement validor client proposals for Deneb (#5094) 2023-06-23 09:30:16 +00:00
Zahary Karadjov b32fac0a25
Merge branch 'stable' into unstable 2023-06-20 02:09:05 +03:00
henridf 0f8866d672
Add Deneb beacon API types (#5060)
* Add Deneb beacon API types

- Introduce the `ProduceBlockResponseV2` type for handling responses to `GET /eth/v2/validator/blocks/{slot}` (https://ethereum.github.io/beacon-APIs/?urls.primaryName=dev#/Validator/produceBlockV2)

- Introduce the `RestPublishedSignedBlockContents` type for handling the request body to `POST /eth/v1/beacon/blocks` (https://ethereum.github.io/beacon-APIs/?urls.primaryName=dev#/ValidatorRequiredApi/publishBlock)

And a few other small changes towards Deneb block production support.

* Address review feedback

* Post-rebase fixes
2023-06-19 08:56:52 +00:00
Etan Kissling 3a6b50cf21
require sync committee supermajority in CI (#5085)
* require sync committee supermajority in CI

To better catch problems with sync committee messages in CI, extend
local testnet simulation to also verify that each block is signed
by a supermajority of the sync committee.

Requires #5083 and #5084

* lint
2023-06-17 06:00:01 +00:00
Etan Kissling aa54760391
avoid implicit `CaseTransition` in keystore Json parsing (#5061)
Override default Json parsing for keystore case objects to avoid
`CaseTransition` logic to be emitted. When parsing the discriminator,
reinitialize the entire object instead of implicitly changing it,
to avoid UB and also possible oversights when extending the object.
See https://github.com/status-im/nim-serialization/pull/59
2023-06-16 19:02:53 +02:00
tersec 591c2246d5
update consensus spec URLs to v1.4.0-alpha.3 (#5088) 2023-06-16 16:45:09 +00:00
henridf ae29babd9f
Implement blob sidecar signatures (#5072)
* Implement blob sidecar signatures

* Update allTests
2023-06-16 15:47:18 +00:00
Jacob Shufro 11f5250ce4
Lower ClientMaximumValidatorIds to match beacon spec limit (#5082) 2023-06-16 09:01:16 +02:00
Etan Kissling a6f0a7a55d
broaden `SszError` catches to `SerializationError` (#5081)
`nim-serialization` is tagged with `{.raises:[SerializationError].}` so
it is no longer sufficient to catch `SszError` in some situations.
`SszError` inherits from `SerializationError`, so broadening the caught
exception types can be done now, to enable bumping `nim-serialization`.

https://github.com/status-im/nimbus-eth2/pull/5043#issuecomment-1584227993

#5061 is also needed to bump `nim-serialization`.
2023-06-15 16:29:13 +00:00
Etan Kissling 142a3e91e5
`ProveField` cleanups in `keystore` (#5050)
Cleanup for `ProveField` warnings in `keystore` module.
Note that `ProveField` is disabled by default in makefile, but sometimes
these pop up when doing a regular `nim c`, and cleaning these may allow
enabling the warning in some future.
2023-06-15 14:53:42 +02:00
tersec ce1503b51f
remove web3signer block signing V1 (phase0 only) support (#5014) 2023-06-15 14:52:19 +02:00
tersec 939b3002e4
remove removed getDebugChainHeadsV1; fix API links in documentations (#5058) 2023-06-15 09:58:38 +00:00
Etan Kissling 411d809c82
split file loading from parsing in helpers (#5054)
* split file loading from parsing in helpers

In `readSszForkedHashedBeaconState` and `readRuntimeConfig`, split the
part that loads the file from the part that parses the file. The parsing
portion can be reused with that, e.g., when loading from the network.

* add missing export marker
2023-06-14 19:48:50 +00:00
Etan Kissling 114453e67d
remove `SAFE_SLOTS_TO_UPDATE_JUSTIFIED` (#5078)
The `SAFE_SLOTS_TO_UPDATE_JUSTIFIED` constant is no longer used in any
EF or Gnosis network metadata definitions. We can remove it.

Upstream changes:

- https://github.com/eth-clients/eth2-networks/pull/87
- https://github.com/gnosischain/configs/pull/10

Which we updated here:

- https://github.com/status-im/nimbus-eth2/pull/5071
- https://github.com/status-im/nimbus-eth2/pull/5077
2023-06-14 18:18:53 +00:00
Eugene Kabanov c0e5c26da1
VC: Use SSZ encoding while processing blocks data (#4999)
* Refactor api.nim to provide more informative failure reasons.
Distinct between unexpected data and unexpected code.
Deprecate Option[T] usage.

* Fix 400 for produceBlindedBlock().
Get proper string conversion for strategy.

* Fix SSZ encoded versions of ProduceBlockResponseV2, ProduceBlockResponseV2 can be received and decoded.
Fix done() warnings.
Bump presto.

* Fix compilation error with new presto.
Use TcpNoDelay option for Web3Signer.

* Fix produceBlockV2() should provide SSZ responses too.

* Address block encoding issue.

* Fix signing test.

* Bump presto.

* Address review comments.
2023-06-14 08:04:15 +02:00
Etan Kissling 6671965fd9
split SSZ and JSON parsing logic in LC handler (#5055)
Use separate functions per format when parsing LC data from REST.
This allows to process events from the eventstream more directly,
as they are always JSON not SSZ. And also makes the code cleaner.
2023-06-13 21:28:17 +00:00
tersec 54cc7bb7a1
update some consensus-spec URLs to v1.4.0-alpha.3 (#5074) 2023-06-13 21:07:39 +00:00
Etan Kissling 20d6b4751e
avoid `ProveField` warning in `keyGen` (#5067)
Assigning to fields of `var` case objects emits `ProveField` warnings.
We disable them in `make` based builds but they may pop up when building
manually with `nim c`. Suppress them for the `keyGen` function, as we
assign to `result.value` separately from `result.ok` to avoid copying.
2023-06-13 16:45:00 +00:00
tersec 788cdb7133
automated v1.4.0-alpha.2 to v1.4.0-alpha.3 consensus spec URL updates (#5065) 2023-06-13 14:03:49 +00:00
tersec d26058b929
use v1.4.0-alpha.3 consensus spec test vectors (#5063) 2023-06-13 10:37:10 +00:00
Etan Kissling 40415ab86c
`ProveField` cleanups in `forks` (#5049)
* `ProveField` cleanups in `forks`

Some more cleanup for `ProveField` warnings in `forks` module.
Note that `ProveField` is disabled by default in makefile, but sometimes
these pop up when doing a regular `nim c`, and cleaning these may allow
enabling the warning in some future.

* use syntax that works if passed to multiple args of call
2023-06-12 12:03:59 +00:00
tersec 22208836b1
automated v1.4.0-alpha.1 to v1.4.0-alpha.2 consensus spec URL updates (#5056) 2023-06-10 09:56:54 +00:00
tersec 1b77831758
Revert "remove removed Debug/getDebugChainHeads V1 (#4970)" (#5057)
This reverts commit 6ebcbbdf4e.
2023-06-10 08:08:01 +00:00
tersec 6ebcbbdf4e
remove removed Debug/getDebugChainHeads V1 (#4970)
* remove removed Debug/getDebugChainHeads V1

* REST API tests
2023-06-10 08:07:36 +00:00
tersec 9e14d904ac
https://github.com/ethereum/consensus-specs/pull/3359 (#5047) 2023-06-10 05:39:10 +00:00
tersec 6a9ed6ddd5
use v1.4.0-alpha.2 consensus spec test vectors (#5051) 2023-06-10 01:45:53 +00:00
Eugene Kabanov effe8b7f90
VC: Remote BN received block monitoring. (#4856)
* Initial commit with both methods enabled: `poll` and `event`.

* Address review comments.

* Address review comments.
Fix copyright years.

* After bump fixes.
2023-06-08 10:44:32 +02:00
henridf 3bda24d4d1
Remove unused param from prepareForkedBlockReading (#5037) 2023-06-07 10:39:04 +00:00
tersec 30c859e8a4
update some consensus-spec URLs to v1.4.0-alpha.1 (#5035) 2023-06-06 17:09:48 +00:00
tersec 58e3de3bf2
add REST call signatures for getStateRandao and getNextWithdrawals (#5033) 2023-06-06 10:01:05 +00:00
Etan Kissling 336acbd39b
display `ExecutionAddress` as hex string (#5029)
When logging `ExecutionAddress`, serialize it as a hex string instead of
as a byte array.
2023-06-04 16:15:14 +00:00
tersec f86febc111
update consensus spec URLs to v1.4.0-alpha.1 (#5027) 2023-06-04 10:32:50 +00:00
tersec b25ca0833b
use v1.4.0-alpha.1 consensus spec test vectors (#5026) 2023-06-03 21:55:08 +00:00
tersec ee71b6cc36
update consensus spec URLs to v1.4.0-alpha.0 (#5022) 2023-06-02 12:59:38 +00:00
Etan Kissling 005a35597f
handle one of the `ProveField` warnings (#5013)
* handle one of the `ProveField` warnings

When assigning between `ForkyHashedBeaconState`, suppress `ProveField`
warning, as `tgt.kind == src.kind` was already checked, but compiler
doesn't understand that (as we only `case tgt.kind`).

* Update beacon_chain/spec/forks.nim

* Update beacon_chain/spec/forks.nim
2023-06-02 01:25:49 +00:00
tersec c9f1bf21d6
refactor state diffs not to require two states in memory (#4986) 2023-05-30 11:55:32 +03:00
tersec 1f535336da
automated v1.3.0 to v1.4.0-alpha.0 consensus spec URL updates (#4996)
* automated v1.3.0 to v1.4.0-alpha.0 consensus spec URL updates

* add copyright year headers
2023-05-26 00:14:28 +02:00
tersec 10569ff2e9
use v1.4.0-alpha.0 consensus spec test vectors (#4990) 2023-05-25 07:55:00 +00:00
Zahary Karadjov a6b3e02ce0
Merge branch 'unstable' into dev/etan/rd-shufflingacc 2023-05-19 14:46:23 +03:00
henridf 01549f6aa4
Wire in blob validation (#4864)
* Wire in blob validation

* Remove unused "is_data_available"

* Log blobs when blob validation fails
2023-05-17 13:55:50 +00:00
Etan Kissling c70fd8fe97
Merge branch 'unstable' into dev/etan/rd-shufflingacc 2023-05-17 14:06:31 +02:00
Etan Kissling 40e89937c5
segregate sync committee messages by period / fork (#4953)
`SyncCommitteeMsgPool` grouped messages by their `beacon_block_root`.
This is problematic around sync committee period boundaries and forks.
Around sync committee period boundaries, members from both the current
and next sync committee may sign the same `beacon_block_root`; mixing
the signatures from both committees together is a mistake. Likewise,
around fork transitions, the `signing_root` changes, so those messages
also need to be segregated.
2023-05-17 07:55:55 +03:00
zah e8572c0246
More relaxed parsing of RemoteKeystores to regain compatibility with version 1 (#4967) 2023-05-16 19:06:16 +03:00
tersec 5aeb5d9209
increase builder API getHeader timeout (#4964) 2023-05-16 16:41:55 +03:00
Jacek Sieka 83393cea8d
dependent slot helpers 2023-05-16 11:04:25 +02:00
Jacek Sieka 3acbb3ea6c
fix missing shuffling dependent slot computation 2023-05-16 10:05:35 +02:00
Etan Kissling adcabf9ad7
cleanup gossip message handler setup (#4961)
* cleanup gossip message handler setup

Reduce duplication and make gossip handler setup easier to read.

* explicit returns for async validators
2023-05-16 09:46:41 +02:00
Etan Kissling dbba003a38
Revert "Revert "accelerate `getShufflingRef` (#4911)" (#4958)"
This reverts commit 748be8b67b.
2023-05-15 17:41:40 +02:00
Etan Kissling 748be8b67b
Revert "accelerate `getShufflingRef` (#4911)" (#4958)
This reverts commit ea97e93e74.
2023-05-15 15:25:51 +00:00
tersec 6024b3e508
rm unused function and unexport some others (#4941) 2023-05-14 22:56:14 +00:00
Etan Kissling 0701038f76
log `bls_to_execution_changes_len` in `shortLog` (#4949)
* log `bls_to_execution_changes_len` in `shortLog`

When logging blocks to console, include num BLS to Execution changes.

* fix

* lint
2023-05-13 04:29:46 +00:00
Etan Kissling ea97e93e74
accelerate `getShufflingRef` (#4911)
When an uncached `ShufflingRef` is requested, we currently replay state
which can take several seconds. Acceleration is possible by:

1. Start from any state with locked-in `get_active_validator_indices`.
   Any blocks / slots applied to such a state can only affect that
   result for future epochs, so are viable for querying target epoch.
   `compute_activation_exit_epoch(state.slot.epoch) > target.epoch`

2. Determine highest common ancestor among `state` and `target.blck`.
   At the ancestor slot, same rules re `get_active_validator_indices`.
   `compute_activation_exit_epoch(ancestorSlot.epoch) > target.epoch`

3. We now have a `state` that shares history with `target.blck` up
   through a common ancestor slot. Any blocks / slots that the `state`
   contains, which are not part of the `target.blck` history, affect
   `get_active_validator_indices` at epochs _after_ `target.epoch`.

4. Select `state.randao_mixes[N]` that is closest to common ancestor.
   Either direction is fine (above / below ancestor).

5. From that RANDAO mix, mix in / out all RANDAO reveals from blocks
   in-between. This is just an XOR operation, so fully reversible.
   `mix = mix xor SHA256(blck.message.body.randao_reveal)`

6. Compute the attester dependent slot from `target.epoch`.
   `if epoch >= 2: (target.epoch - 1).start_slot - 1 else: GENESIS_SLOT`

7. Trace back from `target.blck` to the attester dependent slot.
   We now have the destination for which we want to obtain RANDAO.

8. Mix in all RANDAO reveals from blocks up through the `dependentBlck`.
   Same method, no special handling necessary for epoch transitions.

9. Combine `get_active_validator_indices` from `state` at `target.epoch`
   with the recovered RANDAO value at `dependentBlck` to obtain the
   requested shuffling, and construct the `ShufflingRef` without replay.

* more tests and simplify logic

* test with different number of deposits per branch

* Update beacon_chain/consensus_object_pools/blockchain_dag.nim

Co-authored-by: Jacek Sieka <jacek@status.im>

* `commonAncestor` tests

* lint

---------

Co-authored-by: Jacek Sieka <jacek@status.im>
2023-05-12 19:36:59 +02:00
Etan Kissling d263f7f0cb
fix SSZ response for `produceBlindedBlock` (#4943)
* fix SSZ response for `produceBlindedBlock`

In `produceBlindedBlock`, we sent the `ForkedBlindedBeaconBlock` when
requested to reply in SSZ format. However, expected result is just the
inner `ForkyBlindedBeaconBlock` together with `eth-consensus-version`.

Note: We do not use SSZ format in our VC for this endpoint at this time,
which explains why we haven't noticed earlier.

* fix Altair/Phase0
2023-05-12 15:40:45 +00:00
tersec cc341e0ab7
remove unused definitions (#4937) 2023-05-11 14:39:36 +00:00
Etan Kissling 18bc47fe83
add Capella constants to REST config endpoint (#4925)
Capella constants were missing; if we want to support BLS changes from
VC, we need the new domain constant to check compatibility.
2023-05-11 12:53:02 +02:00
Etan Kissling e0f024c0f3
capella preset for Gnosis (#4936)
```
MAX_WITHDRAWALS_PER_PAYLOAD* = 8
MAX_VALIDATORS_PER_WITHDRAWALS_SWEEP* = 8192
```

https://github.com/gnosischain/configs/pull/11
2023-05-11 09:56:32 +00:00
Etan Kissling 22c298c07b
final portion of trivial v1.3.0 bumps (#4926)
* final portion of trivial v1.3.0 bumps

Updates unchanged logic to latest v1.3.0 consensus-specs refs.

```
https://github.com/ethereum/consensus-specs/(blob|tree)/(?!v1\.3\.0/)
```

* lint
2023-05-11 09:54:29 +00:00
henridf be3f5b1eac
More blob tweaks/fixes from running in devnet (#4933)
* BeaconNode: don't call fetchMissingblobs with empty list

* More logging

* BlockProcessor.checkBloblessSignature: Add missing return value
2023-05-11 00:36:35 +00:00
Etan Kissling 322429b191
final portion of non-trivial v1.3.0 bumps (#4927)
* final portion of non-trivial v1.3.0 bumps

Updates unchanged logic to latest v1.3.0 consensus-specs refs,
and cleans up surrounding sections / syncs comments, and so on.

```
https://github.com/ethereum/consensus-specs/(blob|tree)/(?!v1\.3\.0/)
```

* lint

* linebreak
2023-05-10 16:04:48 +02:00
tersec d3929cbb45
update some beacon API spec URLs; fix some Name and DuplicateModuleImport hints (#4929) 2023-05-10 10:20:55 +00:00
Etan Kissling 4873f8bdc4
update weak subjectivity calculations (#4923)
Weak subjectivity logic (`--weak-subjectivity-checkpoint`) was outdated.
Updated to latest specs, also taking into account total active balance.

See https://github.com/ethereum/consensus-specs/pull/2190
2023-05-10 08:23:59 +00:00
Etan Kissling f4c215c5e6
cleanup `state_transition_epoch` and bump to v1.3.0 (#4922)
* cleanup `state_transition_epoch` and bump to v1.3.0

More v1.3.0 consensus-specs bumps, focused on `state_transition_epoch`.
Also fixed `current_epoch` spurious style check warning, and cleanup.

* Update beacon_chain/spec/state_transition_epoch.nim
2023-05-10 04:31:23 +02:00
Etan Kissling 34e7181b0d
`KZG(Commitment(s)?|Proof)` > `Kzg$1` (#4917)
Fix style-check by using matching capitalization for `nim-kzg` types.
2023-05-09 19:51:36 +00:00
Etan Kissling 93899b2e2a
consensus-specs v1.3.0 bumps with cleanup (#4918)
Some more consensus-specs v1.3.0 bumps. No semantic logic changes,
but also contains cleanup, so separate PR to reduce noise.
2023-05-09 20:16:49 +02:00
Etan Kissling e9319a5947
`bls_to_execution_changes` check alternative (#4916)
Make intent clearer about when to expect `bls_to_execution_changes`,
and replace test-time errors with compile-time errors if the field
is renamed or goes away in the future.
2023-05-09 17:24:02 +00:00
Etan Kissling b04970189b
more v1.3.0 consensus-specs bumps (#4915)
More spec references for unchanged logic bumped to v1.3.0.
2023-05-09 17:20:37 +00:00
zah 5bf9284e62
Initial public version of the Verifying Web3Signer functionality (#4912)
* Allow the list of proved properties for web3signer to be configured
* Document the Web3Signer setups (regular, distributed and verified)
2023-05-09 11:16:43 +03:00
Etan Kissling a23252c297
reorder `get_initial_beacon_block` by fork (#4899)
Deneb was listed between Capella and Bellatrix. Swap with Capella.
2023-05-06 10:32:59 +00:00
Etan Kissling 1ebcd8b473
`excess_data_gas` after `withdrawals` cleanup (#4900)
We already updated the field order in the actual `ExecutionPayload`,
but in init code and tests / logs etc we still used the old order.
Update those occurrences to also match the field order in the struct.
Furthermore, add `excess_data_gas` to last entry in `test_eth1_monitor`.
2023-05-05 23:15:47 +00:00
Etan Kissling 508819633a
more consensus-specs v1.3.0 bumps (#4898)
Updates more consensus-specs references to v1.3.0.
This PR only includes updates for sections otherwise unmodified.
2023-05-05 20:38:51 +00:00
Etan Kissling e408c7573d
bump more consensus-specs refs to v1.3.0 (with comments) (#4901)
Bump some more consensus-specs links to v1.3.0, and also align comments
with latest specs where they differ, and fix typos.
2023-05-05 20:37:56 +00:00
Etan Kissling 28968c11ba
remove `SAFE_SLOTS_TO_UPDATE_JUSTIFIED` (#4897)
The `SAFE_SLOTS_TO_UPDATE_JUSTIFIED` constant is no longer used as the
bouncing attack fix was removed:
https://github.com/ethereum/consensus-specs/pull/3290

Note: Some test networks still define the constant, ignoring the config
constant for now until it is no longer used.
2023-05-05 18:11:38 +00:00
Etan Kissling b5b51de67c
align `BeaconState` docs with latest spec (#4884)
Only comment changes:

- Bump refs to final v1.3.0 spec
- Align documentation style in various `BeaconState` structures
- Add `justification_bits` / `historical_roots` comment from spec
- Remove `previous_justified_checkpoint` from non-phase0 (same as spec)
- Cleanup some `Modified` tags
2023-05-03 13:05:46 +02:00
tersec dc32add555
only trigger liveness failsafe for mainnet (#4882)
* only trigger liveness failsafe for mainnet

* more user-friendly log
2023-05-01 11:45:39 +02:00
henridf ef0b95dfbc
RequestManager: add support for fetching Blobs (#4844)
* RequestManager: add support for fetching Blobs

* Review feedback

* Lint

* Change peekSortedBlobless -> peekBlobless
2023-04-28 12:57:35 +00:00
Etan Kissling dd1ffa5ded
add `el_offline` to `/eth/v1/node/syncing` (#4860)
Add compatibility with https://github.com/ethereum/beacon-APIs/pull/290
to the beacon node. Behaviour when configured with multiple ELs is not
specified; intention suggests to indicate whether all ELs are offline.
2023-04-27 10:47:38 +02:00
Jacek Sieka 58b93ccbe0
era: Capella+ support (fixes #4752) (#4853)
Post-Capella, historical roots are computed from historical summaries
instead of being directly stored in the beacon state.

Slightly messy to pass both lists around - this is done to avoid
computing the historical root unnecessarily.
2023-04-24 15:26:28 +02:00
tersec fbe90dcbea
update statediffs to support capella (#4852) 2023-04-23 22:15:14 +03:00
tersec b390911e93
automatic update of v1.3.0-rc.5 consensus spec URLs to v1.3.0 (#4848) 2023-04-21 18:52:43 +00:00
tersec d3400ca11b
low attestations during epoch should instafail in CI; dbg -> warn level in `newPayload` log (#4830)
* low attestations during epoch should instafail in CI; dbg -> warn level on newPayload log

* improve newPayload warning message when no valid EL connected

* reduce potential spam; make log spelling more consistent; use fatal/quit
2023-04-19 19:42:30 +00:00
tersec 4643a357ef
consensus spec ref URL update to v1.3.0-rc.5 (#4837) 2023-04-18 22:00:06 +00:00
tersec 45f89f1205
use consensus specs v1.3.0 test vectors (#4838) 2023-04-18 22:01:49 +02:00
tersec cc24429828
remove empty block fallback now that capella's on mainnnet (#4821)
* remove empty block fallback now that capella's on mainnnet

* build_empty_execution_payload is only testing infrastructure now

* update error message
2023-04-18 09:21:15 +00:00
tersec 228e10f1d9
update engine API URLs from v1.0.0-beta.2 to beta.3 (#4828) 2023-04-17 20:11:28 +00:00
Jacek Sieka 7df75d77fa
move spec constants to spec (#4825)
* move spec constants to spec

quite funny, this removes >_1000_ transitive imports when building
`nimbus_validator_client`.

`genDepend`:

```sh
[arnetheduck@praeceps beacon_chain]$ diff -u
nimbus_validator_client.dot.pre nimbus_validator_client.dot | wc
   1167    3507  103822
```

* imports

* better constants

* oops

* too many blobs

* reorder spec constants in fork order

* cleanup
2023-04-17 17:18:54 +00:00
Etan Kissling cb9e0eed49
allow trusted node sync based on LC trusted block root (#4736)
* allow trusted node sync based on LC trusted block root

Extends `trustedNodeSync` with a new `--trusted-block-root` option that
allows initializing a light client. No `--state-id` must be provided.
The beacon node will then use this light client to obtain the latest
finalized state from the remote server in a trust-minimized fashion.
Note that the provided `--trusted-block-root` should be somewhat recent,
and that security precautions such as comparing the state root against
block explorers is still recommended.

* fix

* workaround for `valueOr` limitations

* reduce magic numbers

* digest len > context len for readability

* move `cstring` conversion to caller

* avoid abbreviations

* `return` codestyle
2023-04-16 06:07:07 +00:00
Etan Kissling fa3655527c
add `formatIt` for `ForkedLightClientXyz` (#4816)
* add `formatIt` for `ForkedLightClientXyz`

Even though the `ForkyLightClientXyz` have `formatIt`, they do not apply
when logging `ForkedLightClientXyz`, leading to large logs at times.
Defining `formatIt` for `ForkedLightClientXyz` fixes this.

* exports not needed
2023-04-13 15:26:17 +00:00
Etan Kissling dc22244b3e
fix `/eth/v1/debug/fork_choice` output (#4810)
Two fixes to `/eth/v1/debug/fork_choice`:

- `validity` enum is expected to be serialized as string instead of int
- `data` wrapper is not expected for this endpoint
2023-04-11 23:17:48 +00:00
tersec eed34e740a
implement the getStateRandao Beacon API (#4799) 2023-04-11 18:27:48 +03:00
Eugene Kabanov 0ff86e9538
web3signer refactoring and test suite. (#4775)
* Refactor nimbus_signing_node to support Unix signals.

* Fix SN unable to close REST server properly.

* Fix `keys`, `deposit` and `validator_registration` endpoints issues.
Add getValidatorExitSignature() and getDepositMessageSignature() to validator_pool.

* Add /reload endpoint and implementation.
Fix signData to not cancel `timer`.
Fix validator_pool should clear attachedValidators table.

* Diva protocol enhancement implementation.
2023-04-06 16:16:21 +03:00
henridf 04302081b4
Add blob sidecar gossip validation (#4785)
* Add blob gossip validation

* lint

* Add test for getBlobSidecarTopic

* Fix closure variable capture issue

* Update beacon_chain/nimbus_beacon_node.nim

Co-authored-by: tersec <tersec@users.noreply.github.com>

---------

Co-authored-by: tersec <tersec@users.noreply.github.com>
2023-04-04 15:12:34 +00:00
tersec b4508b853a
assorted consensus spec URL updates (#4776) 2023-03-30 23:08:52 +00:00
tersec f71a279d17
more deneb support in ncli_db and forks (#4774) 2023-03-30 10:06:23 +00:00
tersec 0d051b3da8
add Deneb support to REST (de)serialization (#4770) 2023-03-29 11:41:54 +03:00
tersec 9d2fdc2e26
assorted consensus spec URL updates to v1.3.0-rc.5 (#4768) 2023-03-27 14:41:22 +00:00
henridf 635a924e8c
Add nim-kzg4844 and use it in validate_blobs (#4732) 2023-03-23 09:47:04 +00:00
tersec fc1f9a2065
builder API liveness failsafe (#4746)
* builder API liveness failsafe

* add test summary change
2023-03-22 18:48:48 +01:00
tersec 2f634c10a4
automated consensus spec URL updating from v1.3.0-rc.4 to rc.5 (#4756) 2023-03-21 00:42:22 +00:00
tersec 82bf949561
use consensus spec v1.3.0-rc.5 test vectors (#4754) 2023-03-20 20:17:25 +00:00
tersec 0c62950de6
more consensus spec v1.3.0-rc.4 URL updates (#4744) 2023-03-17 12:14:08 +00:00
tersec ec77116414
automated consensus spec URL updating from v1.3.0-rc.3 to rc.4 (#4742) 2023-03-17 01:10:31 +00:00
Etan Kissling fb26e19de9
use v1.3.0-rc.4 consensus spec test vectors (#4740) 2023-03-16 21:35:20 +01:00
Etan Kissling ceefc49965
rm redundant `ConsensusFork.toString` (#4733) 2023-03-14 01:24:10 +00:00
Etan Kissling 29a699f69d
rename `blockFork` > `consensusFork` (#4731)
* rename `blockFork` > `consensusFork`

* revert one overzealous stateFork / blockFork rename
2023-03-12 18:48:38 +00:00
Etan Kissling f0dcbaacff
extend `addTestBlock` with Capella/Deneb support (#4728)
* extend `addTestBlock` with Capella/Deneb support

* bump copyright years

* fix
2023-03-11 20:11:33 +00:00
Etan Kissling fd8e86972d
rename `lcDataForkAtStateFork` > `lcDataForkAtConsensusFork` (#4726) 2023-03-11 20:09:21 +00:00
Etan Kissling 3e872bec86
rename `eip4844_preset.nim` > `deneb_preset.nim` (#4724) 2023-03-11 17:00:35 +00:00
Etan Kissling d8f39dd24f
rename `stateForkForDigest` > `consensusForkForDigest` (#4727) 2023-03-11 16:58:48 +00:00
Etan Kissling b0cccae7f4
rename `(with|at)StateFork` > `(with|at)ConsensusFork` (#4725) 2023-03-11 14:39:29 +00:00
Etan Kissling eba5f9af26
rename `upgrade_to_eip4844` > `upgrade_to_deneb` (#4723) 2023-03-11 14:28:55 +00:00
Etan Kissling 69013d153c
bump light client spec references to `v1.3.0-rc.3` (#4719) 2023-03-11 01:11:51 +00:00
Etan Kissling ad118cd354
rename `stateFork` > `consensusFork` (#4718)
Just the variable, not yet `lcDataForkAtStateFork` / `atStateFork`.

- Shorten comment in `light_client.nim` to keep line width
- Do not rename `stateFork` mention in `runProposalForkchoiceUpdated`.
- Do not rename `stateFork` in `getStateField(dag.headState, fork)`

Rest is just a mechanical mass replace
2023-03-11 00:35:52 +00:00
Etan Kissling 969c6f73ae
misc local `EIP4844` > `Deneb` bumps (#4717)
* misc local `EIP4844` > `Deneb` bumps

* fix
2023-03-11 00:28:19 +00:00
Etan Kissling 57b2151f95
rename `EIP4844` > `Deneb` for light client (#4713)
* rename `EIP4844` > `Deneb` for light client

* regenerate test logs
2023-03-10 17:16:26 +00:00
Etan Kissling e4b5478fff
rename `ForkDigests.eip4844` > `ForkDigests.deneb` (#4715) 2023-03-10 17:13:40 +00:00
tersec a47f0b054e
finish eip4844 to deneb module rename (#4705) 2023-03-09 01:34:17 +01:00
Etan Kissling 879f8ab5e8
check correct fork version in LC sync protocol (#4703)
Tests updated: https://github.com/ethereum/consensus-specs/pull/3284
2023-03-08 19:59:21 +01:00
tersec 63b1b0840f
5 more modules of eip4844.foo to deneb.foo renames (#4698) 2023-03-06 18:45:52 +00:00
zah 8771e91d53
Support for driving multiple EL nodes from a single Nimbus BN (#4465)
* Support for driving multiple EL nodes from a single Nimbus BN

Full list of changes:

* Eth1Monitor has been renamed to ELManager to match its current
  responsibilities better.

* The ELManager is no longer optional in the code (it won't have
  a nil value under any circumstances).

* The support for subscribing for headers was removed as it only
  worked with WebSockets and contributed significant complexity
  while bringing only a very minor advantage.

* The `--web3-url` parameter has been deprecated in favor of a
  new `--el` parameter. The new parameter has a reasonable default
  value and supports specifying a different JWT for each connection.
  Each connection can also be configured with a different set of
  responsibilities (e.g. download deposits, validate blocks and/or
  produce blocks). On the command-line, these properties can be
  configured through URL properties stored in the #anchor part of
  the URL. In TOML files, they come with a very natural syntax
  (althrough the URL scheme is also supported).

* The previously scattered EL-related state and logic is now moved
  to `eth1_monitor.nim` (this module will be renamed to `el_manager.nim`
  in a follow-up commit). State is assigned properly either to the
  `ELManager` or the to individual `ELConnection` objects where
  appropriate.

  The ELManager executes all Engine API requests against all attached
  EL nodes, in parallel. It compares their results and if there is a
  disagreement regarding the validity of a certain payload, this is
  detected and the beacon node is protected from publishing a block
  with a potential execution layer consensus bug in it.

  The BN provides metrics per EL node for the number of successful or
  failed requests for each type Engine API requests. If an EL node
  goes offline and connectivity is resoted later, we report the
  problem and the remedy in edge-triggered fashion.

* More progress towards implementing Deneb block production in the VC
  and comparing the value of blocks produced by the EL and the builder
  API.

* Adds a Makefile target for the zhejiang testnet
2023-03-05 01:40:21 +00:00
tersec e3d96ef147
rename most eip4844Data to denebData (#4693) 2023-03-04 22:23:52 +00:00
tersec 3b41e6a0e7
rename ConsensusFork.EIP4844 to ConsensusFork.Deneb (#4692) 2023-03-04 13:35:39 +00:00
Eugene Kabanov c8b50765cf
Various fixes for VC and BN REST server. (#4673)
* Fix issue when VC unable to detect errors properly and act accordingly.
Switch all API functions used by VC to RestPlainResponse, this allows us to print errors returned by BN servers.

* Fix issue when prepareBeaconCommitteeSubnet() do not perform actions when BN is optimistically synced only.

* Fix Defect issue.

* Fix submit/publish returning `false` when operation was successful.

* Address review comments.

* Fix some client calls unable to receive `execution_optimistic` field, mark BN as OptSynced when such request has been made.

* Adjust warning levels.

---------

Co-authored-by: Jacek Sieka <jacek@status.im>
2023-03-03 20:20:01 +00:00
tersec ea060de6d4
more eip4844 -> deneb module references (#4690) 2023-03-02 21:09:24 +01:00
tersec d058aa09c8
more withdrowls (#4674) 2023-03-02 17:13:35 +01:00
henridf 1de3cf5246
Remove SignedBeaconBlockAndBlobsSidecar (#4683)
This commit removes SignedBeaconBlockAndBlobsSidecar and all remaining
references.
2023-03-02 15:12:04 +01:00
henridf 5a9e63a029
Remove uses of beacon_block_and_blobs_sidecar topic (#4682)
* Remove use of beacon_block_and_blobs_sidecar topic

This topic goes away with decoupled blocks and blobs.

* remove use of getBeaconBlockAndBlobsSidecarTopic from test

* update nimbus_light_client.nim
2023-03-01 21:30:20 +01:00
henridf 3681177cf4
Remove ForkySignedBeaconBlockMaybeBlobs (#4681)
This commit removes ForkySignedBeaconBlockMaybeBlobs and all
references. I tried to pull that thread only as little as was needed
to get rid of it. Left a placeholder BlobSidecar array (in lieu of
Opt[BlobsSidecar]) in a few places; this will be used as we rebuild
the decoupled implementation.
2023-02-28 11:36:17 +00:00
tersec fb1b7458f0
almost all remaining automated v1.3.0-rc.2 to rc.3 consensus spec URL updates (#4679) 2023-02-27 21:37:37 +00:00
tersec 982d79f9a2
more eip4844 -> deneb changes (#4666) 2023-02-25 03:03:34 +02:00
tersec 069d973fc6
use 'deneb' in 4 more modules (#4658) 2023-02-23 19:06:57 +01:00
tersec 8f269c92d7
rename eip48844ImplementationMissing to denebImplementationMissing (#4654) 2023-02-23 10:37:45 +00:00
zah 6036f2e7d7
Local sim impovements (#4551)
* Local sim impovements

* Added support for running Capella and EIP-4844 simulations
  by downloading the correct version of Geth.

* Added support for using Nimbus remote signer and Web3Signer.
  Use 2 out of 3 threshold signing configuration in the mainnet
  configuration and regular remote signing in the minimal one.

* The local testnet simulation can now use a payload builder.
  This is currently not activated in CI due to lack of automated
  procedures for installing third-party relays or builders.

  You are adviced to use mergemock for now, but for most realistic
  results, we can create a simple builder based on the nimbus-eth1
  codebase that will be able to propose transactions from the regular
  network mempool.

* Start the simulation from a merged state. This would allow us
  to start removing pre-merge functionality such as the gossip
  subsciption logic. The commit also removes the merge-forcing
  hack installed after the TTD removal.

* Consolidate all the tools used in the local simulation into a
  single `ncli_testnet` binary.
2023-02-23 02:10:07 +00:00
tersec 1dd07d5def
switch 3 modules in beacon_chain/spec/ from eip4844 to deneb (#4651) 2023-02-22 14:10:00 +01:00
tersec ffaa42b880
fix Deneb/EIP4844 phrasing (#4646) 2023-02-21 19:25:07 +01:00
tersec 29fb65a9db
automated update of v1.3.0-rc.2 to v1.3.0-rc.3 consensus spec URLs (#4647) 2023-02-21 16:43:21 +00:00
tersec 79eddcde40
capella builder API support (#4643)
* capella builder API support

* use capella EPH when appropriate

* fill in ExecutionPayload.withdrawals and sanity-check builder API withdrawals root
2023-02-21 14:21:38 +01:00
tersec 68cb9fe7b1
use consensus spec v1.3.0-rc.3 test vectors (#4645) 2023-02-21 12:58:08 +00:00
tersec b19f7001aa
log expected_withdrawals discrepancy (#4595)
* log expected_withdrawals discrepancy

* capella-only rather than not-bellatrix

* add one more (in theory unnecessary, but) guard against get(none)
2023-02-20 09:46:37 +01:00
tersec 72797cad09
switch some beacon_chain/spec/ from eip4844.Foo to deneb.Foo (#4642) 2023-02-20 09:45:49 +01:00
tersec 0cc0c7e6b7
builder spec v0.3.0 URL updates (#4641)
* builder spec v0.3.0 URL updates

* remove stray non-URL/comment change
2023-02-18 01:54:30 +01:00
tersec a382498cfe
batch-verify BLS to execution change messages (#4637) 2023-02-17 13:35:12 +00:00
tersec dc0bbe3a57
rm blockForkAtEpoch and switch callers to consensusForkAtEpoch (#4634) 2023-02-16 21:16:54 +01:00
Eugene Kabanov e91415662b
Keystore cache implementation. (#4372) 2023-02-16 19:25:48 +02:00
tersec e342fdd97a
rename stateForkAtEpoch to consensusForkAtEpoch (#4627) 2023-02-16 11:32:12 +02:00
zah ff464e49cf
Implement the set of gas_limit end-points in the Keymanager API (#4612)
Fixes #3946
2023-02-15 15:10:31 +00:00
zah 067ba13c52
Capella metadata for Sepolia (#4615)
Other changes:

Renamed the `EIP_4844_FORK_*` config constants to `DENEB_FORK_*` as
this matches the latest spec and it's already used in the official
Sepolia config.
2023-02-15 14:44:09 +00:00
tersec b7c3037f01
consensus spec ref URL updates to v1.3.0-rc.2 (#4626) 2023-02-15 14:43:51 +00:00
Eugene Kabanov 218ea42220
VC: Fix getBlockRoot() response with execution_optimistic. (#4622)
* Fix getStateRoot() and getBlockRoot() API functions which should obtain `execution_optimistic` field.
Fix sync committee service to check `execution_optimistic` field of getBlockRoot() response.

* 2nd part.

* Remove presets usage.
2023-02-15 15:09:31 +01:00
tersec 3977f1529a
fill in remaining capellaImplementationMissing holes for builder API (#4606) 2023-02-14 11:49:48 +01:00
tersec 3011d49946
refactor fcU sending and rename EL-side root to hash (#4614) 2023-02-14 07:48:39 +01:00
tersec 07ccd3fa6e
remove capella and deneb empty execution payload fallbacks (#4613) 2023-02-13 19:15:16 +02:00
tersec d838e5b6ba
remove unused phase0/altair-specific REST deserialization (#4605) 2023-02-13 12:03:38 +01:00
tersec f9022618f0
deprecate req/resp GetMetaData v1 (#4608) 2023-02-13 12:02:20 +01:00
tersec 20fc355ff1
final batch of automated 1.3.0-rc.2 consensus-spec urls (#4604) 2023-02-09 22:08:43 +00:00
tersec ebb45e10da
initial implementation of capella MEV (#4567) 2023-02-06 19:07:30 +01:00
tersec bca781b1b0
prioritize REST API-provided BLS to execution changes over gossip-received changes (#4580) 2023-02-03 16:28:28 +01:00
tersec 1c62a5eb24
capella VC support (#4586) 2023-02-03 16:12:11 +01:00
tersec 956aee2d35
fill some capella/EIP4844 missing implementations (#4585) 2023-02-02 22:24:06 +00:00
tersec eb56762d41
some proc -> func in signatures_batch (#4582) 2023-02-02 19:05:30 +00:00
tersec 63ed5885ab
update engine API URLs to v1.0.0-beta.2 (#4579) 2023-02-01 18:49:36 +00:00
tersec bcc9781cc7
rm obsolete interop module (#4570) 2023-02-01 16:29:55 +01:00
tersec 58ed9308d2
automated v1.3.0-rc.1 to v1.3.0-rc.2 consensus spec URL updates (#4568) 2023-01-31 00:26:57 +01:00
tersec f5e302f973
use 1.3.0-rc.2-hotfix consensus spec test vectors (#4566) 2023-01-30 22:54:16 +00:00
tersec b2f415cbe5
remove merge cruft (#4559) 2023-01-30 20:22:05 +01:00
tersec 8e2792bdd7
remove Nim 1.2 workarounds and `TODO`s (#4533)
* clean up some Nim 1.2 workarounds

* re-add notes about JS backend

* another proc/noSideEffect -> func

* revert ncli/ncli_common.nim changes; 19969 evidently wasn't backported to 1.6
2023-01-30 20:21:51 +01:00
tersec 29ea42ce1a
use v1.3.0-rc.2 consensus spec test vectors (#4564) 2023-01-29 11:28:04 +00:00
tersec 0fb726c420
`BeaconStateFork/BeaconBlockFork` -> `ConsensusFork` (#4560)
* `BeaconStateFork/BeaconBlockFork` -> `ConsensusFork`

* revert unrelated change

* revert unrelated changes

* update test summaries
2023-01-28 19:53:41 +00:00
Eugene Kabanov 08ed8ad43e
Adopt BN and VC header sizes and requirements, to avoid users confusion with default configuration options. (#4556)
Add comments.
2023-01-26 16:00:10 +01:00
tersec c22384d7ba
always use Bellatrix message size limits (#4558) 2023-01-26 14:24:23 +00:00
henridf 7966ab6be2
Some EIP4844 fixes (#4549)
* debug log upon sidecar validation failure

* Fill in signature catch upon SignedBeaconBlockAndBlobsSidecar deser

* Always fill blobssidecar slot and root

* Skip lastFCU when eth1monitor is nil

* fix

* Use cached root
2023-01-25 18:35:46 +01:00
tersec fe1a57c220
use shortLog for execution payload logging (#4544) 2023-01-24 13:19:38 +00:00
Etan Kissling 0b8bb11c50
combine common implementation of LC helpers (#4542)
* combine common implementation of LC helpers

Combine replicated helper code from Altair/Capella/EIP4844 into single
`Forky` based implementation. Also convert `template` to `func` to avoid
selection of incorrect overload.

* fix
2023-01-23 19:38:03 +00:00
henridf 349001b7fb
eip4844 beacon block proposals (#4540)
* eip4844 beacon block proposals

* Don't fetch blobs under minimal preset

@tersec's summary of the issue:

BlobsBundleV1 in the execution API spec assumes a mainnet preset blob
size, where the EIP4844 consensus spec defines
FIELD_ELEMENTS_PER_BLOB: 4 under the minimal preset, which leads to a
Blob having a length of 4 * 32, not 4096 * 32 which BlobsBundleV1
requires.

* Revert unintentional script change
2023-01-21 23:13:21 +00:00
henridf 96dfe18a3d
make message_router.routeSignedBeaconBlock blob-aware (#4537)
* make message_router.outeSignedBeaconBlock blob-aware

* Update beacon_chain/spec/forks.nim

* Update beacon_chain/spec/forks.nim
2023-01-21 06:34:04 +00:00
tersec 8c2e01da24
consensus spec ref URL updates to v1.3.0-rc.1 (#4534) 2023-01-20 22:33:05 +00:00
tersec 6a64048e80
remove vestigial/unused BLSToExecutionChange parameters (#4535) 2023-01-20 20:51:54 +00:00
tersec aacc8d702d
remove Nim 1.2-compatible `push raise`s and update copyright notice years (#4528) 2023-01-20 14:14:37 +00:00
tersec cd58613615
remaining automated rc.0 to rc.1 consensus spec ref URL changes (#4529) 2023-01-20 00:20:51 +00:00
Etan Kissling 65ca523482
support EIP4844 LC data format (#4520)
Implements the proposed light client data format for EIP4844:
- https://github.com/ethereum/consensus-specs/pull/3151
2023-01-19 23:53:37 +01:00
tersec 819e007689
exit/validatorchange pool includes BLS to execution messages; REST support for new pool (#4519)
* exit/validatorchange pool includes BLS to execution messages; REST
support for new pool

* catch failed individual futures

* increase BLS changes bound and keep BLS seen consistent with subpool

* deque capacities should be powers of 2
2023-01-19 22:00:40 +00:00
Etan Kissling 58f4c13e14
support Capella LC data format (#4514)
Implements the proposed light client data format for Capella:
- https://github.com/ethereum/consensus-specs/pull/3151
2023-01-18 15:32:57 +01:00
tersec 073c544f0c
automated update from v1.3.0-rc.0 to v1.3.0-rc.1 consensus spec URLs (#4517) 2023-01-17 16:10:52 +00:00
henridf 727920a571
Refactor block/blobs types (#4491)
* Refactor block/blobs types

Use type system to enforce invariant that a pre-4844 block cannot have
a sidecar.

* Update beacon_chain/nimbus_beacon_node.nim

Co-authored-by: tersec <tersec@users.noreply.github.com>

* review feedback

Co-authored-by: tersec <tersec@users.noreply.github.com>
2023-01-16 16:26:48 +00:00
Etan Kissling fda03548e3
use `ForkedLightClientStore` internally (#4512)
When running `nimbus_light_client`, we persist the latest header from
`LightClientStore.finalized_header` in a database across restarts.
Because the data format is derived from the latest `LightClientStore`,
this could lead to data being persisted in pre-release formats.

To enable us to test later `LightClientStore` versions on devnets,
transition to a `ForkedLightClientStore` internally that is only
migrated to newer forks on-demand (instead of starting at latest).
2023-01-16 16:53:45 +01:00
Etan Kissling 609227559f
LC data fork cleanup (#4506)
Distinguish between those code locations that need to be updated on each
light client data format change, and those others that should generally
be fine, as long as a valid light client object is processed.

The former are tagged with static assert for `LightClientDataFork.high`.

The latter are changed to `lcDataFork > LightClientDataFork.None` to
indicate that they depend only on presence of any valid object.
Also bundled a few minor cleanups and fixes.

Also add `Forky` type for `LightClientStore` and minor fixes / cleanups.
2023-01-14 22:19:50 +01:00
tersec eb02415f00
execution/engine withdrawals amount in uint64 gwei (#4509) 2023-01-14 17:26:57 +00:00
tersec 68e08c6f15
use v1.3.0-rc.1 consensus spec test vectors (#4505) 2023-01-14 04:20:51 +00:00
Etan Kissling 2324136552
add `LightClientHeader` wrapper (#4481)
The light client data structures were changed to accommodate additional
fields in future forks (e.g., to also hold execution data).

There is a minor change to the JSON serialization, where the `header`
properties are now nested inside a `LightClientHeader`.
The SSZ serialization remains compatible.

See https://github.com/ethereum/consensus-specs/pull/3190
and https://github.com/ethereum/beacon-APIs/pull/287
2023-01-13 16:46:35 +01:00
Zahary Karadjov b06502bf65
Gnosis const preset 2023-01-13 04:28:29 +02:00
Etan Kissling 7e276937dc
make LC data fork aware (#4493)
In a future fork, light client data will be extended with execution info
to support more use cases. To anticipate such an upgrade, introduce
`Forky` and `Forked` types, and ready the database schema.
Because the mapping of sync committee periods to fork versions is not
necessarily unique (fork schedule not in sync with period boundaries),
an additional column is added to `period` -> `LightClientUpdate` table.
2023-01-12 18:11:38 +01:00
tersec 4f9064a153
consensus spec ref URL updates (#4496) 2023-01-12 13:33:14 +00:00
henridf 5a03e7400e
Blobs sidecars by range (#4464)
* Add blobs_sidecars_by_range RPC

* Update beacon_chain/spec/datatypes/eip4844.nim

Co-authored-by: tersec <tersec@users.noreply.github.com>

Co-authored-by: tersec <tersec@users.noreply.github.com>
2023-01-12 12:33:24 +00:00
tersec b14106b72b
preserve historical_summaries in capella to eip4844 fork (#4492) 2023-01-11 17:21:16 +00:00
Jacek Sieka ba3db7aa5a
spec: Option -> Opt (#4488) 2023-01-11 12:29:21 +00:00
tersec e28e1aeec8
a few consensus spec ref URL updates (#4483) 2023-01-10 16:14:17 +00:00
Etan Kissling 0590be7afe
add `num_active_participants` helpers for `SyncAggregate` (#4478)
Introduce `num_active_participants` helper function to reduce visibility
of low-level `countOnes` function and reduce code duplication.
2023-01-10 11:26:25 +00:00
tersec 2dd3cd786f
consensus spec ref URL update v1.3.0-{alpha.2,rc.0}; copyright year update (#4477) 2023-01-09 22:44:44 +00:00
tersec 787703bd0c
use v1.3.0-rc.0 consensus spec test vectors (#4472) 2023-01-09 15:15:43 +01:00
Jacek Sieka 0ba9fc4ede
History pruning (fixes #4419) (#4445)
Introduce (optional) pruning of historical data - a pruned node will
continue to answer queries for historical data up to
`MIN_EPOCHS_FOR_BLOCK_REQUESTS` epochs, or roughly 5 months, capping
typical database usage at around 60-70gb.

To enable pruning, add `--history=prune` to the command line - on the
first start, old data will be cleared (which may take a while) - after
that, data is pruned continuously.

When pruning an existing database, the database will not shrink -
instead, the freed space is recycled as the node continues to run - to
free up space, perform a trusted node sync with a fresh database.

When switching on archive mode in a pruned node, history is retained
from that point onwards.

History pruning is scheduled to be enabled by default in a future
release.

In this PR, `minimal` mode from #4419 is not implemented meaning
retention periods for states and blocks are always the same - depending
on user demand, a future PR may implement `minimal` as well.
2023-01-07 10:02:15 +00:00
tersec 47cb0f7991
capella forkchoiceUpdated support (#4462)
* capella forkchoiceUpdated support

* match V2 fcU with V2 getPayload
2023-01-06 22:01:10 +01:00
tersec c5d1683f19
spec ref URL & copyright year updates (#4467) 2023-01-06 16:28:46 +00:00
henridf 8251cc223d
eip4844 gossip (#4444)
* eip4844 gossip

* Check BLSFieldElement range validity in gossip validation

* lint/nits cleanup

* Use template to avoid an assignment with copy.

* More review feedback

* lint

* lint

* processSignedBeaconBlockAndBlobsSidecar: clean up error handling flow

* Undo factoring-out of beacon blocks validator installation
2023-01-04 12:34:15 +00:00
Jacek Sieka aff0505807
Allow reindexing to start from an arbitrary state (#4437)
When not backfilling all the way to genesis (#4421), it becomes more
useful to start rebuilding the historical indices from an arbitrary
starting point.

To rebuild the index from non-genesis, a state and an unbroken block
history is needed - here, we allow loading the state from an era file
and recreating the history from there onwards.

* speed up partial era state loading
2023-01-03 20:37:09 +01:00
tersec 45654984a9
capella validator withdrawal credentials aren't immutable (#4455) 2023-01-03 20:04:59 +01:00
Jacek Sieka d854317646
fix countdown over time period (#4449)
*sigh*

fixes:
```
Error: unhandled exception:
beacon_chain/beacon_chain_db_light_client.nim(214, 12)
`period.isSupportedBySQLite`  [AssertionDefect]
```
2022-12-30 21:36:31 +01:00
Etan Kissling 2184fd2322
bump `nim-eth`, extend empty block fallback for EIP4844 (#4425)
Implements `emptyPayloadToBlockHeader` for EIP-4844.
https://github.com/status-im/nim-eth/pull/570
2022-12-20 20:00:56 +01:00
Etan Kissling c91d9d61e2
validate EL block hash in EL simulation (#4420)
When simulating EL with `--optimistic` flag, perform block hash check.
2022-12-20 09:24:33 +01:00
zah 07d4160e00
Migrating the deposit contract snapshot can no longer fail on start-up (#4438)
The missing piece of data that had to be obtained previously from
the configured EL client is now part of the network metadata baked
into the binary.
2022-12-19 18:19:48 +01:00
Jacek Sieka bd8f08204e
Implement skip_randao_verification for blinded blocks (#4435)
* Implement skip_randao_verification for blinded blocks

* fix redundant randao verification on block replay (5% faster)
* check randao in REST instead of internally
* avoid redundant copies when making blocks
* cleanup leftover randao skipping code

* fix test summary
2022-12-19 15:11:12 +02:00
Jacek Sieka 7501f10587
60% state replay speedup (#4434)
* 60% state replay speedup

* don't use HashList for epoch participation - in addition to the code
currently clearing the caches several times redundantly, clearing has to
be done each block nullifying the benefit (35%)
* introduce active balance cache - computing it is slow due to cache
unfriendliness in the random access pattern and bounds checking and we
do it for every block - this cache follows the same update pattern as
the active validator index cache (20%)
* avoid recomputing base reward several times per attestation (5%)

Applying 1024 blocks goes from 20s to ~8s on my laptop - these kinds of
requests happen on historical REST queries but also whenever there's a
reorg.

* fix test and diffs
2022-12-19 14:01:49 +02:00
tersec 110401394b
consolidate consensus spec transition test fixtures (#4428)
* consolidate consensus spec transition test fixtures

* include capella

* consoliate fork test fixtures

* note change in EIP-4844 process_block in alpha.2
2022-12-15 22:51:04 +00:00
tersec bb4ea37baa
update EF consensus spec URLs from v1.3.0-alpha.1 to v1.3.0-alpha.2 (#4432) 2022-12-15 12:15:12 +00:00
tersec d269d82b71
use v1.3.0-alpha.2 EF consensus spec test vectors (#4430) 2022-12-15 01:51:06 +00:00
tersec 7faef7827e
fix EIP4844 withBlck (#4411)
* fix EIP4844 withBlck

* don't raiseAssert by default
2022-12-14 18:30:56 +01:00
tersec cd993ca418
consolidate slot sanity checks across forks (#4418)
* consolidate slot sanity checks across forks

* use `new` rather than `init` for constructor of `ref`s
2022-12-13 15:58:59 +00:00
tersec bc996623e0
add EIP4844 block database read/write test (#4416) 2022-12-13 00:56:50 +00:00
tersec e3c062189a
eip4844 `block_sim` (#4405) 2022-12-09 21:39:11 +00:00
tersec 94653b0b71
fix bellatrix web3signer client-side beacon block signing requests (#4407) 2022-12-08 21:57:47 +00:00
tersec dee5af58d6
eip4844 light client tests; avoid case object out-of-bound array reads (#4404) 2022-12-08 17:21:53 +01:00
tersec 50bcc48e17
eip4844 block/slot sanity and transition tests (#4399) 2022-12-08 02:07:41 +00:00
tersec 2932d3b808
extent `BeaconStateFork` enum (#4396) 2022-12-07 16:47:23 +00:00
zah d30cb8baf1
Support for obtaining deposit snapshots during trustedNodeSync (#4303)
Other changes:

* More optimal search for TTD block.

* Add timeouts to all REST requests during trusted node sync.
  Fixes #4037

* Removed support for storing a deposit snapshot in the network
  metadata.
2022-12-07 12:24:51 +02:00
tersec 7cf432b155
eip4844 fork and epoch transition tests; some eip4844 gossip (#4393) 2022-12-06 16:43:11 +00:00
tersec 031780b612
eip4844 operations tests (#4392) 2022-12-06 12:40:13 +00:00
Eugene Kabanov a311f04a19
Implementation for REST /eth/v1/validator/liveness/{epoch} endpoint. (#4381)
Switch VC to use /eth/v1/validator/liveness/{epoch} endpoint instead of nimbus one.
2022-12-06 12:29:00 +01:00
tersec 415b11aa67
EIP4844 tweaks to pass SSZ consensus object tests (#4390) 2022-12-05 21:36:53 +00:00
henridf f0329b2212
Types and scaffolding for EIP-4844 (#4365)
* Types and scaffolding for EIP-4844

This commit adds the EIP-4844 spec types, and fills in
scaffolding/boilerplate for the use of these types across the repo.

None of the actual EIP-4844 logic is introduced yet.

This follows the pattern used by @tersec when introducing Capella (#4276).

* use eth2-networks fork

* review feedback: add static check EIP4844_FORK_EPOCH == FAR_FUTURE_EPOCH

* review feedback: remove EIP4844 from /eth/v1/config/spec response

* Cleanup / review feedback

* Fix REST test
2022-12-05 16:29:09 +00:00
tersec 38827d776c
capella gossip support (#4386) 2022-12-04 08:42:03 +01:00
tersec 5c16062de9
remove all but truly stub support for `SHARDING_FORK_{EPOCH,VERSION}` (#4385) 2022-12-02 13:33:18 +01:00
tersec 4e71e77da7
structure for supporting capella block production (#4383) 2022-12-02 08:39:01 +01:00
tersec c0cacd2561
use safer case object transitions (#4379) 2022-12-01 14:36:44 +02:00
zah 7c783644a2
Fix and document some usages of defaultRuntimeConfig (#4147)
Other changes:

* Make the light client store compatible with phase0-only networks
  and simulations
2022-12-01 11:25:21 +00:00
tersec 474b0d8502
`withUpdatedState` injects `updatedState` rather than `state` template (#4375) 2022-11-30 16:37:23 +02:00
tersec df54470b13
verify `CAPELLA_FORK_EPOCH == FAR_FUTURE_EPOCH` for supported networks (#4369) 2022-11-29 09:12:25 +01:00
tersec ed672113bc
support engine API execution payloads with withdrawals (#4358) 2022-11-29 05:02:16 +00:00
tersec cf58fa9573
remove unused types from datatypes/bellatrix and capella (#4366) 2022-11-28 16:30:38 +00:00
Etan Kissling c941dc801f
bump `nim-eth`, extend empty block fallback for Capella (#4357)
Implements `emptyPayloadToBlockHeader` for Capella.
https://github.com/status-im/nim-eth/pull/562
2022-11-28 14:41:25 +01:00
tersec 898684c175
REST server shouldn't create validation objects when optimistically synced (#4361) 2022-11-25 22:49:31 +00:00
tersec 61c5ac32d8
automated consensus spec ref URL update to v1.3.0-alpha.1 (#4354) 2022-11-24 19:07:02 +00:00
tersec 1fceb33b2e
more capella (#4350) 2022-11-24 14:38:07 +00:00
Eugene Kabanov 8fa6064b9a
VC: blinded block publishing support (#4332)
* Add blind REST API declarations and implementations.

* shortLog is still not stable.

* Fix shortLog issues.

* Enable disabled logging statements.

* Address review comments.

* Avoid templates suffering from double evaluation of their params

* Address review comments.

* Fix compilation issue.

Co-authored-by: Zahary Karadjov <zahary@status.im>
2022-11-24 09:14:05 +00:00
tersec c8083f2c32
implement more missing capella functionality (#4344) 2022-11-24 09:53:04 +02:00
tersec ec443601eb
implement capellaImplementationMissing points; don't track not-active validator duties (#4340)
* implement several capellaImplementationMissing points

* don't register validator activity for not-active validators

* don't check validator indices already coming out of committees which exist; must be active validators, or else other deeper bugs
2022-11-22 13:56:05 +02:00
tersec 1146470f7d
use v1.3.0-alpha.1 consensus spec test vectors (#4338) 2022-11-21 08:44:49 +01:00
tersec b3f6be71d5
refactor `makeBeaconBlock`; some capella support for `ncli_db` and `wss_sim` (#4321) 2022-11-11 15:37:43 +01:00
tersec 35b1104bea
`block_sim` runs capella by default (#4315) 2022-11-11 10:17:27 +00:00
tersec 04cbea754b
don't require attached validator for blinded block BN endpoint (#4313) 2022-11-10 20:18:08 +00:00
Etan Kissling 48994f67d3
rename `BlockError` -> `VerifierError` (#4310)
We currently use `BlockError` for both beacon blocks and LC objects.
In light of EIP4844, we will likely also use it for blob sidecars.
To avoid confusion, renaming it to a more generic `VerifierError`,
and update its documentation to be more generic.

To avoid long lines as a followup, also renaming the `block_processor`'s
`BlockProcessingCompleted.completed`->`ProcessingStatus.completed` and
`BlockProcessingCompleted.notCompleted`->`ProcessingStatus.notCompleted`
2022-11-10 17:40:27 +00:00
Jacek Sieka 09ade6d33d
Make trusted node sync era-aware (#4283)
This PR removes a bunch of code to make TNS aware of era files, avoiding
a duplicated backfill when era files are available.

* reuse chaindag for loading backfill state, replacing the TNS homebrew
* fix era block iteration to skip empty slots
* add tests for `can_advance_slots`
2022-11-10 10:44:47 +00:00
Jacek Sieka 1691d89695
3x key loading speedup via blst sha256 (refs #4277) (#4301)
* 3x key loading speedup via blst sha256 (refs #4277)

* readd sha2
2022-11-10 00:05:50 +02:00
tersec 90eb2ccb20
database and fork choice test runner support for capella (#4309) 2022-11-09 17:32:10 +00:00
tersec 0919b8689e
run capella fork transition tests in CI (#4307) 2022-11-09 12:28:34 +00:00
tersec 5c0ad1b5c8
automated v1.2.0 -> v1.3.0-alpha.0 consensus spec URL update (#4305) 2022-11-09 09:20:53 +00:00
tersec a4d2637259
implement /eth/v1/beacon/blinded_blocks BN endpoint (#4286)
* implement /eth/v1/beacon/blinded_blocks BN endpoint

* avoid function names in non-debug logs

* callers log, so callee can just return err()
2022-11-08 18:08:43 +00:00
Etan Kissling 7ad610f6d7
set fee recipient in empty payload fallback (#4291)
When the EL/Builder fails to produce an execution payload, we fall back
to an empty `ExecutionPayload`. Even though it contains no transactions
it should refer to the configured fee recipient. This is useful for
privacy reasons (do not reveal the reason for the empty payload) and for
compliance with additional fee recipient rules by staking pools.
2022-11-08 14:19:56 +00:00
tersec 909c095e64
initial automated v1.2.0 -> v1.3.0-alpha.0 consensus spec URL update (#4296) 2022-11-08 02:37:28 +00:00
Jacek Sieka fae3ed5067
log block number / fee recipient (#4294) 2022-11-07 22:16:47 +00:00
tersec 0a43c89cd2
run capella block sanity tests in CI (#4292) 2022-11-07 18:37:48 +00:00
tersec 5b46f0b723
add Capella support to Forked* (#4276)
* add Capella support to Forked*

* remove cruft

* add `OnForkyBlockAdded`
2022-11-02 16:23:30 +00:00
tersec 3ef09ff596
Support `/eth/v1/validator/blinded_blocks` (#4272)
* Support BN endpoints for producing blinded blocks

* use correct endpoint version

* serve either JSON or SSZ versions of endpoint
2022-10-31 18:39:03 +01:00
Eugene Kabanov 00f083785d
VC: Fix for #4116 (external block builders support) (#4260) 2022-10-29 11:00:51 +02:00
tersec 0cfc1b776e
add all missing epoch transition tests (#4269) 2022-10-28 08:02:33 +00:00
tersec 7dd5c49c4e
use v1.3.0-alpha.0 test vectors (#4263) 2022-10-27 11:54:39 +00:00
tersec 06ccf5b80c
capella test vector support (#4261) 2022-10-27 06:29:24 +00:00
Jacek Sieka af52edaef7
Reduce fee recipient log spam (#4255)
Once is enough
2022-10-25 14:47:43 +02:00
tersec f9830836a9
deprecate --terminal-total-difficulty-override; remove launch script for deprecated ropsten (#4241)
* deprecate --terminal-total-difficulty-override; remove launch script for deprecated ropsten

* remove Makefile support for Ropsten
2022-10-24 23:32:52 +03:00
tersec fb6e6d9cf4
remove `newPayload` from block production flow (#4186)
* remove `newPayload` from block production flow

* refactor block_processor to run `newPayload` as part of `storeBlock`
2022-10-14 22:48:56 +03:00
Zahary Karadjov 89582c10db
Merge branch 'stable' into unstable 2022-10-14 22:24:55 +03:00
cheatfate b54a03ae07
Fix MEV builder file descriptor leaks. 2022-10-14 16:09:39 +03:00
Etan Kissling 95aec9de12
update to v1 of LC REST API (#4232)
The LC REST API has been merged into the ethereum/beacon-APIs specs:
- https://github.com/ethereum/beacon-APIs/pull/247

Update URLs to v1 and update REST tests. Note that REST tests do not
start with Altair, so the tested BN will return empty / error responses.
2022-10-13 13:42:53 +03:00
Etan Kissling 4b7bb4796f
update for latest LC REST proposal (#4213)
Implements the latest proposal for providing LC data via REST, as of
https://github.com/ethereum/beacon-APIs/pull/247 with a v0 suffix.

Requests:
- `/eth/v0/beacon/light_client/bootstrap/{block_root}`
- `/eth/v0/beacon/light_client/updates?start_period={start_period}&count={count}`
- `/eth/v0/beacon/light_client/finality_update`
- `/eth/v0/beacon/light_client/optimistic_update`

HTTP Server-Sent Events (SSE):
- `light_client_finality_update_v0`
- `light_client_optimistic_update_v0`
2022-10-13 13:42:46 +03:00
Etan Kissling 3a4b87358a
update to v1 of LC REST API (#4232)
The LC REST API has been merged into the ethereum/beacon-APIs specs:
- https://github.com/ethereum/beacon-APIs/pull/247

Update URLs to v1 and update REST tests. Note that REST tests do not
start with Altair, so the tested BN will return empty / error responses.
2022-10-13 02:16:49 +02:00
tersec 16817fef95
cleanups: `proc` -> `func`, unused import, spec URLs (#4224) 2022-10-08 05:07:54 -05:00
tersec ce915c0a03
add beaconcha.in mainnet relay overview to guide (#4217) 2022-10-04 19:45:06 +03:00
Etan Kissling 7202acddff
update for latest LC REST proposal (#4213)
Implements the latest proposal for providing LC data via REST, as of
https://github.com/ethereum/beacon-APIs/pull/247 with a v0 suffix.

Requests:
- `/eth/v0/beacon/light_client/bootstrap/{block_root}`
- `/eth/v0/beacon/light_client/updates?start_period={start_period}&count={count}`
- `/eth/v0/beacon/light_client/finality_update`
- `/eth/v0/beacon/light_client/optimistic_update`

HTTP Server-Sent Events (SSE):
- `light_client_finality_update_v0`
- `light_client_optimistic_update_v0`
2022-10-04 13:38:09 +02:00
Etan Kissling 83cd104d07
add section headers for REST serialization (#4210)
Adds missing REST serialization sections and makes them consistent.
2022-10-03 22:29:07 +00:00
Etan Kissling 231eb8a590
use same style for sending block and state rsps (#4209)
For JSON responses, "eth-consensus-version" header is handled in
`eth2_rest_serialization` for states and `rest_beacon_api` for blocks.
Align them to also be handled in `eth2_rest_serialization` for blocks.
2022-10-04 01:05:52 +03:00
tersec ad7541567c
move LVH handling to tests/; increase maximum fork choice retries (#4205) 2022-10-03 13:10:08 +00:00
tersec 0a4aa5fdb3
switch `withStateAndBlck` usage to `forkyState`; rm unused `tests/mocking/` modules (#4206) 2022-10-03 13:08:50 +00:00
Eugene Kabanov a845450283
REST API: add ssz encoding for publishBlock (#4154) 2022-09-30 00:00:53 +03:00
Jacek Sieka af9ec577d0
nicer error message for failed backfill (#4188)
* nicer error message for failed backfill

Many checkpoint sources don't support block download

* RestGenericError -> RestErrorMessage

...and other assorted fixes to bring rest types closer to spec

* fix tests
2022-09-29 23:55:18 +03:00
tersec c367b14ad9
deprecate `--safe-slots-to-import-optimistically` (#4182) 2022-09-29 06:29:49 +00:00
Eugene Kabanov 8778e1cf8d
Fix REST generic error parsing. (#4189)
* Fix REST generic error parser.
* Unescape test vectors.
* Fix RestGenericError writer and tests, to encode `code` as `Number`.
2022-09-28 18:47:15 +00:00
Jacek Sieka c11b30f8e1
Remove deprecated REST API calls from client (#4190)
* https://github.com/ethereum/beacon-APIs/pull/220
* https://github.com/ethereum/beacon-APIs/pull/218
2022-09-28 10:37:54 +00:00
tersec 57d68d0f72
re-enable randao checks (#4187)
* re-enable randao checks

* use `asSigVerified` consistently

* fix spelling

* document why state_transition.makeBeaconBlock trusting signatures is safe
2022-09-28 01:15:10 +00:00
tersec a0ead042ad
newPayload `INVALIDATED` should be `unviableFork` (#4180) 2022-09-26 21:24:32 +00:00
tersec 9750cd3a38
update state diffs to Bellatrix (#4177) 2022-09-26 19:13:50 +00:00
Etan Kissling d76df065a1
rm unused `dump` function for `KeystoresAndSlashingProtection` (#4170)
In `eth2_rest_serialization` there was a `dump` function for
`KeystoresAndSlashingProtection` that does not seem to be used.
Removes that unused function.
2022-09-26 21:09:51 +02:00
Etan Kissling 7ac95c6ee7
move `EpochParticipationFlags` serialization to others (#4167)
The REST `writeValue/readValue` serialization functions are grouped up
with the exception of `EpochParticipationFlags`. Moving that to others.
2022-09-23 18:29:31 +00:00
Etan Kissling 77ea188c92
add `sszResponsePlain` for pre-encoded SSZ data (#4171)
For pre-encoded JSON REST responses we have `jsonResponsePlain`.
Adds a `sszResponsePlain` function to serve similar purpose for SSZ.
This avoids caller having to explicitly specify Http200 and media type.
2022-09-23 15:51:04 +00:00
Etan Kissling 8b11778e14
use `withState` in `jsonResponseState` (#4169)
The generic sandwich preventing `forks.withState` from working in REST
serialization is no longer applicable. Update to use `withState`.
2022-09-23 13:49:22 +00:00
Etan Kissling cd626cf645
delete unused REST version constants (#4166)
Removes unused `Phase0Version` and `AltairVersion` constants.
2022-09-23 12:48:53 +00:00
tersec deb043796b
a few more manual v1.2.0 consensus spec ref URL updates (#4165) 2022-09-23 12:00:17 +00:00
tersec 3c03ba86c1
update consensus spec ref URLs to v1.2.0 (#4164) 2022-09-23 07:56:06 +00:00
tersec 72e6b2021a
use v1.2.0 consensus spec test vectors (#4163) 2022-09-22 22:24:13 +00:00
Michael Sproul d6325b1ab5
api: add skip_randao_verification for produceBlockV2 (#3837) 2022-09-21 09:38:08 +02:00
tersec 56720dd808
update consensus layer spec ref URLs to v1.2.0-rc.3 (#4143) 2022-09-20 02:08:09 +02:00
zah ad63bba446
Support Prysm and Ethdo Keystores (Fixes #4107) (#4149) 2022-09-20 01:09:56 +03:00
tersec e3750e96e8
fix order of current/previous dependent root in REST SSE (#4141) 2022-09-19 13:28:52 +03:00
Eugene Kabanov ca871a5435
Fix HTTP/REST clients HTTP Content-Type header parsers. (#4139)
* Fix client HTTP content-type parsers.

* Fix tests.

* Address review comment and apply wildcard checks for generic decodeBytes.
2022-09-19 12:17:29 +03:00
Etan Kissling 9999362b11
detect mismatch of config and binary (#4132)
* detect mismatch of config and binary

When loading configuration that sets keys that Nimbus bakes into the
binary at compile-time, raise an error if the config is incompatible
instead of ignoring the conflicting value.
2022-09-19 12:07:46 +03:00
Jacek Sieka ef8bab58eb
load suggested fee recipient file also when keymanager is disabled (#4078)
Since these files may have been created in a previous run or manually,
we want to keep loading them even on nodes that don't enable the
keystore API (for example static setups)

Other changes:

* log keystore loading progressively (#3699)
* print initial fee recipient when loading validators
* log dynamic fee recipient updates
2022-09-17 08:30:07 +03:00
tersec 0410aec9d8
remove rest of `withState.state` usage (#4120)
* remove rest of `withState.state` usage

* remove scaffolding
2022-09-16 15:35:00 +02:00
Jacek Sieka 43188a0990
clean up exchange configuration handling (#4126)
Per spec, we should not be sending our detected terminal block to EL -
the EL configuration exchange should only look at values from
configuration and report mismatches.
2022-09-16 15:33:22 +02:00
tersec 80f44f4491
update consensus layer spec ref URLs to v1.2.0-rc.3 (#4117) 2022-09-13 17:30:11 +00:00
tersec 02a99543c6
more `withState` `state` -> `forkyState` (#4112) 2022-09-13 14:53:12 +03:00
tersec 8be964a152
update consensus layer spec ref URLs to v1.2.0-rc.3 (#4109) 2022-09-10 17:16:38 +00:00
tersec 19bf460a3b
more `withState` `state` -> `forkyState` (#4104) 2022-09-10 08:12:07 +02:00
tersec 1d620f0123
consensus spec URL updates to v1.2.0-rc.3 (#4105) 2022-09-09 21:56:06 +00:00
Etan Kissling 0191225896
improve deposit processing performance (#4082)
When there are a lot of deposits, we decompress the public key into a
crypto cache. To avoid having those caches grow unreasonably big,
make sure to operate on the decompressed pubkey instead.
2022-09-07 20:49:32 +02:00
Jacek Sieka dee60e6e77
avoid rechecking bellatrix+ block signatures in state transition (#4085)
Like the other forks, these are already batch-verified - this also
avoids polluting the last-resort global key cache
2022-09-07 16:45:46 +00:00
Jacek Sieka d9ceb61dbd
eth: bump (#4062) 2022-09-04 19:44:43 +02:00
tersec 301e5a919d
remove some Bellatrix-specific references (#4019)
* remove some Bellatrix-specific references

* remove more bellatrixData-dependencies
2022-09-03 20:56:20 +00:00
tersec ad0d30093f
state/forkyState cleanup; spec URL updates; rm unused imports (#4052) 2022-08-31 13:29:34 +02:00
tersec 9ae796daed
Cache and resend, rather than recreate, builder API registrations (#4040) 2022-08-31 03:29:03 +03:00
tersec b60456fdf3
`withState`: `state` -> `forkyState` (#4038) 2022-08-26 22:47:40 +00:00
tersec 66a5e88203
allow accessing withState forky state via `forkyState` (#4026) 2022-08-26 17:14:18 +03:00
tersec 61dc296046
update engine API spec ref URLs from alpha.9 to beta.1 (#4030)
* update engine API spec ref URLs from alpha.9 to beta.1

* require exactly 256-bit JWT keys
2022-08-26 13:44:50 +03:00
Etan Kissling 9180f09641
reduce LC optsync latency (#4002)
The optimistic sync spec was updated since the LC based optsync module
was introduced. It is no longer necessary to wait for the justified
checkpoint to have execution enabled; instead, any block is okay to be
optimistically imported to the EL client, as long as its parent block
has execution enabled. Complex syncing logic has been removed, and the
LC optsync module will now follow gossip directly, reducing the latency
when using this module. Note that because this is now based on gossip
instead of using sync manager / request manager, that individual blocks
may be missed. However, EL clients should recover from this by fetching
missing blocks themselves.
2022-08-25 03:53:59 +00:00
tersec e70d5e6194
update spec ref URLs in state_transition_epoch (#4016) 2022-08-23 13:06:12 +00:00
Etan Kissling f1ddcfff0f
support connecting to peers without bellatrix (#4011)
* support connecting to peers without bellatrix

Make discovery fork ID aware of scheduled Bellatrix fork to enable
connections to peers that don't have Bellatrix scheduled yet.
Without this, has peering issues with peers on older SW version.

* expand tests with compatibility checks

* more exhaustive compatibility checks
2022-08-21 19:36:46 +02:00
tersec c65eaca1bf
update spec ref URLs (#4005) 2022-08-20 16:03:32 +00:00
zah b1ac9c9fe4
Fix a potential segfault and various potential stalls (#4003)
* Fixes a segfault during block production when the Keymanager API
  is disabled. The Keymanager is now disabled on half of the local
  testnet nodes to catch such problems in the future.

* Fixes multiple potential stalls from REST requests being done
  without a timeout. From practice, we know that such requests
  can hang forever if not cancelled with a timeout. At best,
  this would be a resource leak, at worst, it may lead to a
  full stall of the client and missed validator duties.

* Changes some Options usages to Opt (for easier use of valueOr)
2022-08-19 21:51:30 +00:00
zah fca20e08d6
Keymanager API for the validator client (#3976)
* Keymanager API for the validator client
* Properly treat the 'description' field as optional when loading Keystores
* Spec-compliant serialization of the slashing data in Keymanager's DeleteKeys response ()

Fixes #3940
Fixes #3964
Closes #3884 by adding test
2022-08-19 13:30:07 +03:00
zah a7192f5d6c
Fix the block header computation when proposing an empty execution payload (#3991)
* Fix the block header computation when proposing an empty execution payload
* Spec compliant base fee calculation when producing empty payloads
2022-08-19 13:28:42 +03:00
tersec b5b93e90c0
use v1.2.0-rc.3 test vectors (#3995) 2022-08-19 04:32:53 +00:00
tersec c0f673dc09
spec ref URL updates: v1.2.0-rc.{1,2} for phase0/fork-choice altair/beacon-chain (#3986) 2022-08-18 07:25:33 +00:00
Etan Kissling 5c8e58ea23
update LC spec references for v1.2.0-rc.2 (#3982)
Updates light client spec references for latest spec (no more `vFuture`)
2022-08-17 19:47:06 +00:00
tersec 8274d5373b
update spec ref URLs (#3979) 2022-08-17 11:33:19 +00:00
tersec e732b080f7
use v1.2.0-rc.2 test vectors (#3975) 2022-08-16 21:24:37 +00:00
zah 06a5c67e62
The body of prepare_beacon_proposer is an array (Fixes #3920) (#3938) 2022-08-09 12:53:02 +03:00
Eugene Kabanov 250f7b4bdf
Exclusive keystore locking (#3907) 2022-08-08 00:53:20 +03:00
Eugene Kabanov fe5435ed96
Address #3624 (#3934) 2022-08-06 11:55:40 +00:00
tersec d62d13a23c
MEV block proposal (#3883)
* MEV validator registration

* add nearby canary to detect new beacon chain forks

* remove special MEV graffiti

* web3signer support

* fix trace logging

* Nim 1.2 needs raises Defect

* use template rather than proc in REST JSON parsing

* use --payload-builder-enable and --payload-builder-url

* explicitly default MEV to disabled

* explicitly empty default value for payload builder URL

* revert attestation pool to unstable version
2022-08-01 09:41:47 +03:00
Miran dfd4afc9f2
compatibility with Nim 1.4+ (#3888) 2022-07-29 10:53:42 +00:00
Etan Kissling 3ec7982293
update light client protocol version (#3550)
* Use final `v1` version for light client protocols
* Unhide LC data collection options
* Default enable LC data serving
* rm unneeded import
* Connect to EL on startup
* Add docs for LC based EL sync
2022-07-29 11:45:39 +03:00
Zahary Karadjov 4637669b6b
Merge branch 'stable' into unstable 2022-07-26 23:54:46 +03:00
zah cd04f27c37
Implement the /eth/v1/validator/prepare_beacon_proposer end-point (#3901) 2022-07-25 23:12:53 +03:00
Etan Kissling 3bc42994e4
update to latest LC test format (#3879)
The EF test format for the LC sync protocol is modified to verify checks
after each step: https://github.com/ethereum/consensus-specs/pull/2938 -
The test runner is updated accordingly.
2022-07-23 05:54:01 +00:00
Eugene Kabanov c3d3397843
VC: doppelganger protection (#3877)
* Improve fallback_service.

* Improve logging in fallback_service.

* Apply signal handling for all stages.

* Fix some logging statements.

* Add doppelganger REST api endpoint.
Add some structures to VC.

* Add client API call implementation.

* Initial fix & refactor onceToAll()
Add doppelganger service.
Add doppelganger helpers.

* Add doppelganger checks.

* Move doppelganger log messages to higher levels.

* Fix firstSuccess().

* Bump chronos.

* Post rebase fixes.

* Proper chronos bump.

* Address review comments.

* Attempt to fix finalization test issue.

* Fix nimbus_signing_node.

* Mark validators which are added at GENESIS_SLOT in GENESIS_EPOCH as passed doppelganger validation.

* Do not send empty requests to server.

* Fix log statement.

* Address review comments and re-raise cancellations.

Co-authored-by: zah <zahary@gmail.com>
2022-07-21 19:54:07 +03:00
Etan Kissling d6adf09e65
cleanup magic strings (#3891)
Replaces hardcoded IP address strings with constants.
Adjusts some LC log levels.
Removes no longer used `RpcServer` type.
2022-07-20 18:17:21 +00:00
zah 20d45e69b5
Re-enabled requireAllFields after a fix in nim-json-serialization (#3871)
* Re-enabled requireAllFields after a fix in nim-json-serialization

The problem was that `Option[T]` fields were not treated as optional
when requireAllFields is set to true. This is now fixed in NJS.

* Add makefile targets for recreating the Jenkins simulation runs

* Fix a discrepancy with the REST spec
2022-07-15 03:19:19 +03:00
Etan Kissling 47a1b11d5d
revert `requireAllFields` regression (#3870)
#3864 introduced a regression by turning on `requireAllFields` globally
for JSON parsing. Certain endpoints such as `RestSyncInfo` have optional
fields that do not parse correctly without additional changes. This is
reverted for now to restore previous behaviour and unblock CI testing.
2022-07-14 03:33:15 +00:00
zah 806536a040
[Keymanager API] Support for the feerecipient end-points (#3864)
Other changes:

* The Keymanager error responses differ from the Beacon API responses.
  'keymanagerApiError' replaces the former usages of 'jsonError'.

* Return status code 401 and 403 for authorization errors in accordance
  to the spec.

* Eliminate inconsistencies in the REST JSON parsing. Some of the code
  paths allowed missing fields.

* Added logging of serialization failure details at DEBUG level.
2022-07-13 17:45:04 +03:00
tersec ce6cbd84e2
rename verifyFinalization internal flag to strictVerification (#3866)
* rename verifyFinalization internal flag to strictVerification

* Update beacon_chain/extras.nim

Co-authored-by: Etan Kissling <etan@status.im>

Co-authored-by: Etan Kissling <etan@status.im>
2022-07-13 13:48:09 +00:00
tersec e0e7af7aff
MEV boost script for ropsten and sepolia builder network (#3851)
* MEV boost script for ropsten and sepolia builder network

* networkInfo not a global variable
2022-07-12 20:50:12 +03:00
Etan Kissling f9536fd14b
fork choice cleanups (#3849)
Removes a few extra-ambitious templates to make `self` updates explicit,
and moves the `FinalityCheckpoints` type from `base` to `helpers` as it
is an additional Nimbus specific type not defined by spec.
2022-07-10 15:26:29 +00:00
Jacek Sieka e1830519a4
Introduce message router (#3829)
Whether new blocks/attestations/etc are produced internally or received
via REST, their journey through the node is the same - to ensure that
they get the same treatment (logging, metrics, processing), this PR
moves the routing to a dedicated module and fixes several small
differences that existed before.

* `xxxValidator` -> `processMessageName` - the processor also was adding
messages to pools, so we want the name to reflect that action
* add missing "sent" metrics for some messages
* document ignore policy better - already-seen messages are not actaully
rebroadcast by libp2p
* skip redundant signature checks for internal validators consistently
2022-07-06 16:11:44 +00:00
Etan Kissling 2a2bcea70d
group justified and finalized `Checkpoint` (#3841)
The justified and finalized `Checkpoint` are frequently passed around
together. This introduces a new `FinalityCheckpoint` data structure that
combines them into one.

Due to the large usage of this structure in fork choice, also took this
opportunity to update fork choice tests to the latest v1.2.0-rc.1 spec.
Many additional tests enabled, some need more work, e.g. EL mock blocks.
Also implemented `discard_equivocations` which was skipped in #3661,
and improved code reuse across fork choice logic while at it.
2022-07-06 13:33:02 +03:00
tersec 19bf7f1599
check in MEV boost test script that payload gets unblinded (#3838) 2022-07-05 04:13:07 +00:00
tersec 1221bb66e8
optimistic sync (#3793)
* optimistic sync

* flag that initially loaded blocks from database might need execution block root filled in

* return optimistic status in REST calls

* refactor blockslot pruning

* ensure beacon_blocks_by_{root,range} do not provide optimistic blocks

* handle forkchoice head being pre-merge with block being postmerge

* re-enable blocking head updates on validator duties

* fix is_optimistic_candidate_block per spec; don't crash with nil future

* fix is_optimistic_candidate_block per spec; don't crash with nil future

* mark blocks sans execution payloads valid during head update
2022-07-04 23:35:33 +03:00
Jacek Sieka c145916414
cleanups (#3819)
* avoid circular panda imports
* move deposit merkleization helpers to spec/
* normalize validator signature helpers to spec names / params
* remove redundant functions for remote signing
2022-06-29 18:53:59 +02:00
Etan Kissling bc1cc8f643
encapsulate LC config into one type (#3817)
Separate LC initialization options from the main ChainDAGRef options to
allow ChainDAGRef to treat them as opaque and reduce risk for conflicts
when extending those options in the future.
2022-06-28 22:52:29 +02:00
Etan Kissling e8e9ce1aab
introduce types for LC merkle proofs (#3808)
Merkle proofs tend to have long underlying type definitions, e.g.,
`array[log2trunc(NEXT_SYNC_COMMITTEE_INDEX), Eth2Digest]`. For the
ones used in the LC sync protocol, dedicated types are introduced
to improve readability. Furthermore, the `CachedLightClientBootstrap`
wrapper that solely wrapped a merkle branch is eliminated.
2022-06-28 07:52:23 +02:00
Etan Kissling 91d543440a
add option to configure max historic LC data periods (#3799)
Adds a `--light-client-data-max-periods` option to override the number
of sync committee periods to retain light client data.
Raising it above the default enables archive nodes to serve full data.
Lowering below the default speeds up import times (still no persistence)
2022-06-27 13:24:38 +02:00
Etan Kissling aa1b8e4a17
bump nim-ssz-serialization to `3db6cc0f282708aca6c290914488edd832971d61` (#3119)
This updates `nim-ssz-serialization` to
`3db6cc0f282708aca6c290914488edd832971d61`.

Notable changes:
- Use `uint64` for `GeneralizedIndex`
- Add support for building merkle multiproofs
2022-06-26 19:33:06 +02:00
tersec 56cd6e2b45
rm unused imports and fix stylecheck issues (#3784) 2022-06-21 21:01:45 +02:00
Jacek Sieka 347a485b5b
bearssl: split abi (#3755) 2022-06-21 10:29:16 +02:00
tersec 7e61198260
fix merge test scripts (#3781) 2022-06-20 18:03:57 +00:00
Eugene Kabanov eb6b7affee
Add the `execution_optimistic` flag to REST API responses. (#3780)
* Initial commit

* Make `events` API spec compliant.

* Add `Eth-Consensus-Version` in responses.

* Bump chronos to get redirect with headers working.

* Add `is_optimistic` field and handling to syncing RestSyncInfo.
2022-06-20 08:53:39 +03:00
zah c24c737866
Fix #3650 (participation format in BeaconState result is out of spec) (#3776)
* Fix #3650 (participation format in BeaconState result is out of spec)
* Make EpochParticipationFlags a distinct type
2022-06-20 08:38:56 +03:00
zah 5402dfab6b
Correct URLs for the DeleteKeys request in the Keymanager API (#3727)
Other changes:

* Make it easier to run the REST tests locally through a Makefile target
2022-06-19 20:54:12 +03:00
Eugene Kabanov c4383e21f4
Fix #3760. (#3768)
* Address #3760.

Add test vectors for deserialization.
2022-06-19 09:08:21 +03:00
Etan Kissling 61ee0611bb
adopt LC REST API with v0 suffix (without proofs) (#3775)
* adopt LC REST API with v0 suffix (without proofs)

Adopts the light client data REST API used by Lodestar as defined in
https://github.com/ethereum/beacon-APIs/pull/181 with a v0 suffix.

Requests:
- `/eth/v0/beacon/light_client/bootstrap/{block_root}`
- `/eth/v0/beacon/light_client/updates?start_period={start_period}&count={count}`
- `/eth/v0/beacon/light_client/finality_update`
- `/eth/v0/beacon/light_client/optimistic_update`

HTTP Server-Sent Events (SSE):
- `light_client_finality_update_v0`
- `light_client_optimistic_update_v0`

More work is needed to adopt the proofs endpoint, it is not included.

* initialize event queues

* register event topics
2022-06-19 08:57:52 +03:00
tersec cbb0b8142f
include capella fork version in fork consistency check (#3772)
* include capella fork version in fork consistency check

* Update tests/test_conf.nim

Co-authored-by: Etan Kissling <etan@status.im>

Co-authored-by: Etan Kissling <etan@status.im>
2022-06-18 10:05:33 +00:00
tersec 2924ed9604
remove some usage of calls not required by engine API (#3761) 2022-06-17 06:32:52 +00:00
zah e8efc0f184
Add support for the Sepolia network (#3762) 2022-06-16 17:11:26 +03:00
Etan Kissling 15967c4076
keep track of latest blocks for optimistic sync (#3715)
When launched with `--light-client-enable` the latest blocks are fetched
and optimistic candidate blocks are passed to a callback (log for now).
This helps accelerate syncing in the future (optimistic sync).
2022-06-10 14:16:37 +00:00