Commit Graph

1350 Commits

Author SHA1 Message Date
Etan Kissling eb3a30655b
generalize `ShufflingRef` acceleration logic (#5197)
Split up the `ShufflingRef` acceleration logic into generically usable
parts and attester shuffling specific parts. The generic parts could be
used to accelerate other purposes, e.g., REST `/states/xxx/randao` API.
2023-07-20 10:25:39 +02:00
Etan Kissling da5bb96856
avoid Nim 2.0 issue when loading config (#5202)
To avoid Nim 2.0 issue https://github.com/nim-lang/Nim/issues/22284,
explicitly specify `ref SecondarySources` instead of using `auto`,
and add `{.raises.}` annotation
2023-07-19 20:26:30 +02:00
Etan Kissling f98c33ad03
generalize `commonAncestor` function to `BlockId` (#5192)
To enable additional use cases, e.g., `/states/###/randao` beacon API,
`ShufflingRef` acceleration logic needs to be able to operate on parts
of the DAG that do not have `BlockRef`. Changing `commonAncestor` to
act on `BlockId` instead of `BlockRef` is a step toward that and also
simplifies the logic some more.
2023-07-18 17:37:53 +02:00
Etan Kissling 2efc44a8ab
accelerate RANDAO computation for post-merge blocks (#5190)
Post-merge blocks contain all information to directly obtain RANDAO
without having to load any additional info. Take advantage of that to
further accelerate `ShufflingRef` computation. Note that it is still
necessary to verify that `blck` / `state` share a sufficiently recent
ancestor for the purpose of computing attester shufflings.

- new: 243.71s, 239.67s, 237.32s, 238.36s, 239.57s
- old: 251.33s, 234.29s, 249.28s, 237.03s, 236.78s
2023-07-15 22:16:56 +02:00
Etan Kissling f1963bf599
allow initializing test DBs from later forks (#5186)
Running `makeTestDB` in tests currently always initializes DB with a
`phase0` state, preventing tests that configure a fork schedule that
starts in a different fork from working properly. Fix that by upgrading
the genesis state to whatever fork the fork schedule starts with.
2023-07-14 09:35:58 +02:00
Etan Kissling 74bb4b1411
simplify RANDAO recovery in `ShufflingRef` acceleration (#5183)
Current RANDAO recovery logic is quite complex as it optimizes for the
minimum amount of database reads. Loading blocks isn't the bottleneck
though, so rather make the implementation more concise by avoiding the
complex strategy planning step. Note that this also prepares for an even
faster implementation for post-merge blocks in the future that extracts
RANDAO from `ExecutionPayload` directly if available, so even in cases
where efficiency is slightly lower, only historical data is affected.

`time nim c -r tests/test_blockchain_dag` (cached binary):

- new: 145.45s, 133.59s, 144.65s, 127.69s, 136.14s
- old: 149.15s, 150.84s, 135.77s, 137.49s, 133.89s
2023-07-12 17:27:05 +02:00
Jacek Sieka ca1775f725
Fetch by-root request directly from quarantine (#5167)
When the requestmanager is busy fetching blocks, the queue might get
filled with multiple entries of the same root - since there is no
deduplication, requests containing the same root multiple times will be
sent out.

Also, because the items sit in the queue for a long time potentially,
the request might be stale by the time that the manager is ready with
the previous request.

This PR removes the queue and directly fetches the blocks to download
from the quarantine which solves both problems (the quarantine already
de-duplicates and is clean of stale information).

Removing the queue for blobs is left for a future PR.

Co-authored-by: tersec <tersec@users.noreply.github.com>
2023-07-11 18:22:02 +02:00
Etan Kissling 5115aaedb7
early exit `commonAncestor` when comparing with `finalizedHead` (#5174)
* early exit `commonAncestor` when comparing with `finalizedHead`

As all `BlockRef` lead to `finalizedHead` (`parent == nil`),
can shortcut in that situation and immediately return `finalizedHead`
if passed as one of the arguments.

* typo in comment

* add test from #5152

Co-authored-by: tersec <tersec@users.noreply.github.com>

* add note about test complexity

* regenerate test summary

---------

Co-authored-by: tersec <tersec@users.noreply.github.com>
2023-07-10 20:36:25 +00:00
Eugene Kabanov ac1b02698a
VC: Use scoring function to select best attestation data when using multiple BNs. (#5101)
* Initial commit.

* Move score selection log statement to debug level.

* Fix proper float64 log format.

* Cleanup imports and legacy code.

* Address review comments.

* Address review comments.

* Fix scoring function.

* Address review comments.

* Address review comments 2.
Fix registerBlock post-rebase issues.

* Simplify innerLoop decision making.

* Make getAttestationDataScore() more testable.
Add tests for getAttestationDataScore().

* Add modified AllTests copy.
2023-07-06 11:14:22 +02:00
tersec ba94dc849f
opt-out support for v1.4.0 attestation stability subnets (#5128)
* opt-out support for v1.4.0 attestation stability subnets

* document debug-prefix-flag-policy and remove temporary from flag name
2023-06-30 05:23:15 +00:00
tersec d5a6d51ede
Web3Signer support only for only Bellatrix and later (#5107)
Co-authored-by: Etan Kissling <etan@status.im>
2023-06-29 09:36:43 +00:00
henridf 1234900065
Add blob handling to message router (#5106)
* Add blob handling to message router

* address review feedback

* Fix typos
2023-06-28 17:55:31 +00:00
tersec 614202e30d
automated consensus spec URL updating to v1.4.0-beta.0 (#5121) 2023-06-24 15:43:30 +00:00
tersec b62664915e
consensus spec v1.4.0 attestation stability subnets (#5092) 2023-06-23 09:30:46 +00:00
Etan Kissling 2722778ce5
reduce `nim-eth` dependencies just for RNG (#5099)
We have several modules that import `nim-eth` for the sole purpose of
its `keys.newRng` function. This function is meanwhile a simple wrapper
around `nim-bearssl`'s `HmacDrbgContext.new()`, so the import doesn't
really serve a use anymore. Replace `keys.newRng` with the direct call
to reduce `nim-eth` imports.
2023-06-19 22:43:50 +00:00
henridf 0f8866d672
Add Deneb beacon API types (#5060)
* Add Deneb beacon API types

- Introduce the `ProduceBlockResponseV2` type for handling responses to `GET /eth/v2/validator/blocks/{slot}` (https://ethereum.github.io/beacon-APIs/?urls.primaryName=dev#/Validator/produceBlockV2)

- Introduce the `RestPublishedSignedBlockContents` type for handling the request body to `POST /eth/v1/beacon/blocks` (https://ethereum.github.io/beacon-APIs/?urls.primaryName=dev#/ValidatorRequiredApi/publishBlock)

And a few other small changes towards Deneb block production support.

* Address review feedback

* Post-rebase fixes
2023-06-19 08:56:52 +00:00
Etan Kissling 3a6b50cf21
require sync committee supermajority in CI (#5085)
* require sync committee supermajority in CI

To better catch problems with sync committee messages in CI, extend
local testnet simulation to also verify that each block is signed
by a supermajority of the sync committee.

Requires #5083 and #5084

* lint
2023-06-17 06:00:01 +00:00
Etan Kissling 8c6c8a0ffa
use correct slot when producing sync aggregate around forks (#5089)
`produceSyncAggregate` is called in new slot when block is produced,
while the other functions in `sync_committee_msg_pool` are called in
previous slot. So, need to subtract 1 slot when producing sync aggregate
to accept the signatures using the old digest during fork transition.
2023-06-16 21:30:36 +00:00
tersec 591c2246d5
update consensus spec URLs to v1.4.0-alpha.3 (#5088) 2023-06-16 16:45:09 +00:00
henridf ae29babd9f
Implement blob sidecar signatures (#5072)
* Implement blob sidecar signatures

* Update allTests
2023-06-16 15:47:18 +00:00
tersec ce1503b51f
remove web3signer block signing V1 (phase0 only) support (#5014) 2023-06-15 14:52:19 +02:00
Eugene Kabanov c0e5c26da1
VC: Use SSZ encoding while processing blocks data (#4999)
* Refactor api.nim to provide more informative failure reasons.
Distinct between unexpected data and unexpected code.
Deprecate Option[T] usage.

* Fix 400 for produceBlindedBlock().
Get proper string conversion for strategy.

* Fix SSZ encoded versions of ProduceBlockResponseV2, ProduceBlockResponseV2 can be received and decoded.
Fix done() warnings.
Bump presto.

* Fix compilation error with new presto.
Use TcpNoDelay option for Web3Signer.

* Fix produceBlockV2() should provide SSZ responses too.

* Address block encoding issue.

* Fix signing test.

* Bump presto.

* Address review comments.
2023-06-14 08:04:15 +02:00
tersec 788cdb7133
automated v1.4.0-alpha.2 to v1.4.0-alpha.3 consensus spec URL updates (#5065) 2023-06-13 14:03:49 +00:00
tersec 22208836b1
automated v1.4.0-alpha.1 to v1.4.0-alpha.2 consensus spec URL updates (#5056) 2023-06-10 09:56:54 +00:00
tersec 30c859e8a4
update some consensus-spec URLs to v1.4.0-alpha.1 (#5035) 2023-06-06 17:09:48 +00:00
Etan Kissling 12381d5f33
also pack attestations where LMD vote is orphaned (#5002)
* also pack attestations where LMD vote is orphaned

When `attestation.data.beacon_block_root` gets orphaned, attestations
with a good `attestation.data.target.root` may still be valuable.
The LMD GHOST vote is not relevant for attestation rewards.

Switch to use the FFG vote (`attestation.data.target.root`) instead,
gossip validation ensures it is an ancestor of `beacon_block_root`.

* lint
2023-06-04 10:39:12 +00:00
tersec f86febc111
update consensus spec URLs to v1.4.0-alpha.1 (#5027) 2023-06-04 10:32:50 +00:00
tersec b25ca0833b
use v1.4.0-alpha.1 consensus spec test vectors (#5026) 2023-06-03 21:55:08 +00:00
Etan Kissling d7890ac013
avoid port number conflict in `test_signing_node` (#5018)
Allow multiple copies of `test_signing_node` to run concurrently by
ensuring that they use unique network port numbers.
2023-06-01 09:33:25 +00:00
tersec 7a7573d7d0
fix chronos done -> completed deprecation warnings (#5016) 2023-06-01 10:04:30 +02:00
tersec c9f1bf21d6
refactor state diffs not to require two states in memory (#4986) 2023-05-30 11:55:32 +03:00
tersec 1f535336da
automated v1.3.0 to v1.4.0-alpha.0 consensus spec URL updates (#4996)
* automated v1.3.0 to v1.4.0-alpha.0 consensus spec URL updates

* add copyright year headers
2023-05-26 00:14:28 +02:00
tersec 10569ff2e9
use v1.4.0-alpha.0 consensus spec test vectors (#4990) 2023-05-25 07:55:00 +00:00
henridf 5ef748b19d
Clarify addOrphan error/logging (#4981)
* Clarify addOrphan error/logging

addOrphan returned a bool to indicate success. Change this to a Result
so that different errors can be distinguished.

* Update beacon_chain/consensus_object_pools/block_quarantine.nim

Co-authored-by: tersec <tersec@users.noreply.github.com>

* Update beacon_chain/gossip_processing/gossip_validation.nim

---------

Co-authored-by: tersec <tersec@users.noreply.github.com>
2023-05-21 17:47:00 +00:00
henridf 1cf777c64b
Fix sync for blocks older than MIN_EPOCHS_FOR_BLOB_SIDECARS_REQUESTS (#4977)
When doing sync for blocks older than
MIN_EPOCHS_FOR_BLOB_SIDECARS_REQUESTS, we skip the blobs by range
request, but we then pass en empty blob sequence to
validation, which then fails.

To fix this: Use an Option[Blobsidecars] to allow expressing the
distinction between "empty blob sequence" and "blobs unavailable". Use
the latter for "old" blocks, and don't attempt to run blob validation.
2023-05-19 16:25:11 +00:00
Zahary Karadjov a6b3e02ce0
Merge branch 'unstable' into dev/etan/rd-shufflingacc 2023-05-19 14:46:23 +03:00
Zahary Karadjov 4c3850f7df
Disable the use of incbin as it breaks the retail build on Linux/ARM 2023-05-19 04:08:02 +03:00
cheatfate a36cacda44
New stricter beacon-node URL parsing 2023-05-19 02:02:20 +03:00
Etan Kissling c70fd8fe97
Merge branch 'unstable' into dev/etan/rd-shufflingacc 2023-05-17 14:06:31 +02:00
tersec 74511f61d1
Use withdrawal credentials as default fee recipient (#4968) 2023-05-17 07:56:37 +03:00
Etan Kissling 40e89937c5
segregate sync committee messages by period / fork (#4953)
`SyncCommitteeMsgPool` grouped messages by their `beacon_block_root`.
This is problematic around sync committee period boundaries and forks.
Around sync committee period boundaries, members from both the current
and next sync committee may sign the same `beacon_block_root`; mixing
the signatures from both committees together is a mistake. Likewise,
around fork transitions, the `signing_root` changes, so those messages
also need to be segregated.
2023-05-17 07:55:55 +03:00
zah e8572c0246
More relaxed parsing of RemoteKeystores to regain compatibility with version 1 (#4967) 2023-05-16 19:06:16 +03:00
Jacek Sieka 83393cea8d
dependent slot helpers 2023-05-16 11:04:25 +02:00
Etan Kissling dbba003a38
Revert "Revert "accelerate `getShufflingRef` (#4911)" (#4958)"
This reverts commit 748be8b67b.
2023-05-15 17:41:40 +02:00
Etan Kissling 748be8b67b
Revert "accelerate `getShufflingRef` (#4911)" (#4958)
This reverts commit ea97e93e74.
2023-05-15 15:25:51 +00:00
henridf 573228ffa0
Rename eth1/ -> el/ and eth1_monitor.nim -> el_monitor.nim (#4944) 2023-05-15 05:05:12 +00:00
tersec 10a7281349
don't run flaky CI in Windows (#4935)
* don't run flaky CI in Windows

* reduce stack size in `test_light_client`

* regen tests

* re-enable maybe non-problematic test module on Windows

---------

Co-authored-by: Etan Kissling <etan@status.im>
2023-05-15 04:14:58 +02:00
Etan Kissling ec049bc493
use correct root in `test_gossip_validation` (#4951)
Use block root not state root for sync committee message in test.
2023-05-14 16:18:50 +02:00
Etan Kissling ea97e93e74
accelerate `getShufflingRef` (#4911)
When an uncached `ShufflingRef` is requested, we currently replay state
which can take several seconds. Acceleration is possible by:

1. Start from any state with locked-in `get_active_validator_indices`.
   Any blocks / slots applied to such a state can only affect that
   result for future epochs, so are viable for querying target epoch.
   `compute_activation_exit_epoch(state.slot.epoch) > target.epoch`

2. Determine highest common ancestor among `state` and `target.blck`.
   At the ancestor slot, same rules re `get_active_validator_indices`.
   `compute_activation_exit_epoch(ancestorSlot.epoch) > target.epoch`

3. We now have a `state` that shares history with `target.blck` up
   through a common ancestor slot. Any blocks / slots that the `state`
   contains, which are not part of the `target.blck` history, affect
   `get_active_validator_indices` at epochs _after_ `target.epoch`.

4. Select `state.randao_mixes[N]` that is closest to common ancestor.
   Either direction is fine (above / below ancestor).

5. From that RANDAO mix, mix in / out all RANDAO reveals from blocks
   in-between. This is just an XOR operation, so fully reversible.
   `mix = mix xor SHA256(blck.message.body.randao_reveal)`

6. Compute the attester dependent slot from `target.epoch`.
   `if epoch >= 2: (target.epoch - 1).start_slot - 1 else: GENESIS_SLOT`

7. Trace back from `target.blck` to the attester dependent slot.
   We now have the destination for which we want to obtain RANDAO.

8. Mix in all RANDAO reveals from blocks up through the `dependentBlck`.
   Same method, no special handling necessary for epoch transitions.

9. Combine `get_active_validator_indices` from `state` at `target.epoch`
   with the recovered RANDAO value at `dependentBlck` to obtain the
   requested shuffling, and construct the `ShufflingRef` without replay.

* more tests and simplify logic

* test with different number of deposits per branch

* Update beacon_chain/consensus_object_pools/blockchain_dag.nim

Co-authored-by: Jacek Sieka <jacek@status.im>

* `commonAncestor` tests

* lint

---------

Co-authored-by: Jacek Sieka <jacek@status.im>
2023-05-12 19:36:59 +02:00
Jacek Sieka b3c6320d56
embed genesis states using incbin (#4905) 2023-05-11 11:11:00 +00:00