Commit Graph

1343 Commits

Author SHA1 Message Date
Etan Kissling 5115aaedb7
early exit `commonAncestor` when comparing with `finalizedHead` (#5174)
* early exit `commonAncestor` when comparing with `finalizedHead`

As all `BlockRef` lead to `finalizedHead` (`parent == nil`),
can shortcut in that situation and immediately return `finalizedHead`
if passed as one of the arguments.

* typo in comment

* add test from #5152

Co-authored-by: tersec <tersec@users.noreply.github.com>

* add note about test complexity

* regenerate test summary

---------

Co-authored-by: tersec <tersec@users.noreply.github.com>
2023-07-10 20:36:25 +00:00
Eugene Kabanov ac1b02698a
VC: Use scoring function to select best attestation data when using multiple BNs. (#5101)
* Initial commit.

* Move score selection log statement to debug level.

* Fix proper float64 log format.

* Cleanup imports and legacy code.

* Address review comments.

* Address review comments.

* Fix scoring function.

* Address review comments.

* Address review comments 2.
Fix registerBlock post-rebase issues.

* Simplify innerLoop decision making.

* Make getAttestationDataScore() more testable.
Add tests for getAttestationDataScore().

* Add modified AllTests copy.
2023-07-06 11:14:22 +02:00
tersec ba94dc849f
opt-out support for v1.4.0 attestation stability subnets (#5128)
* opt-out support for v1.4.0 attestation stability subnets

* document debug-prefix-flag-policy and remove temporary from flag name
2023-06-30 05:23:15 +00:00
tersec d5a6d51ede
Web3Signer support only for only Bellatrix and later (#5107)
Co-authored-by: Etan Kissling <etan@status.im>
2023-06-29 09:36:43 +00:00
henridf 1234900065
Add blob handling to message router (#5106)
* Add blob handling to message router

* address review feedback

* Fix typos
2023-06-28 17:55:31 +00:00
tersec 614202e30d
automated consensus spec URL updating to v1.4.0-beta.0 (#5121) 2023-06-24 15:43:30 +00:00
tersec b62664915e
consensus spec v1.4.0 attestation stability subnets (#5092) 2023-06-23 09:30:46 +00:00
Etan Kissling 2722778ce5
reduce `nim-eth` dependencies just for RNG (#5099)
We have several modules that import `nim-eth` for the sole purpose of
its `keys.newRng` function. This function is meanwhile a simple wrapper
around `nim-bearssl`'s `HmacDrbgContext.new()`, so the import doesn't
really serve a use anymore. Replace `keys.newRng` with the direct call
to reduce `nim-eth` imports.
2023-06-19 22:43:50 +00:00
henridf 0f8866d672
Add Deneb beacon API types (#5060)
* Add Deneb beacon API types

- Introduce the `ProduceBlockResponseV2` type for handling responses to `GET /eth/v2/validator/blocks/{slot}` (https://ethereum.github.io/beacon-APIs/?urls.primaryName=dev#/Validator/produceBlockV2)

- Introduce the `RestPublishedSignedBlockContents` type for handling the request body to `POST /eth/v1/beacon/blocks` (https://ethereum.github.io/beacon-APIs/?urls.primaryName=dev#/ValidatorRequiredApi/publishBlock)

And a few other small changes towards Deneb block production support.

* Address review feedback

* Post-rebase fixes
2023-06-19 08:56:52 +00:00
Etan Kissling 3a6b50cf21
require sync committee supermajority in CI (#5085)
* require sync committee supermajority in CI

To better catch problems with sync committee messages in CI, extend
local testnet simulation to also verify that each block is signed
by a supermajority of the sync committee.

Requires #5083 and #5084

* lint
2023-06-17 06:00:01 +00:00
Etan Kissling 8c6c8a0ffa
use correct slot when producing sync aggregate around forks (#5089)
`produceSyncAggregate` is called in new slot when block is produced,
while the other functions in `sync_committee_msg_pool` are called in
previous slot. So, need to subtract 1 slot when producing sync aggregate
to accept the signatures using the old digest during fork transition.
2023-06-16 21:30:36 +00:00
tersec 591c2246d5
update consensus spec URLs to v1.4.0-alpha.3 (#5088) 2023-06-16 16:45:09 +00:00
henridf ae29babd9f
Implement blob sidecar signatures (#5072)
* Implement blob sidecar signatures

* Update allTests
2023-06-16 15:47:18 +00:00
tersec ce1503b51f
remove web3signer block signing V1 (phase0 only) support (#5014) 2023-06-15 14:52:19 +02:00
Eugene Kabanov c0e5c26da1
VC: Use SSZ encoding while processing blocks data (#4999)
* Refactor api.nim to provide more informative failure reasons.
Distinct between unexpected data and unexpected code.
Deprecate Option[T] usage.

* Fix 400 for produceBlindedBlock().
Get proper string conversion for strategy.

* Fix SSZ encoded versions of ProduceBlockResponseV2, ProduceBlockResponseV2 can be received and decoded.
Fix done() warnings.
Bump presto.

* Fix compilation error with new presto.
Use TcpNoDelay option for Web3Signer.

* Fix produceBlockV2() should provide SSZ responses too.

* Address block encoding issue.

* Fix signing test.

* Bump presto.

* Address review comments.
2023-06-14 08:04:15 +02:00
tersec 788cdb7133
automated v1.4.0-alpha.2 to v1.4.0-alpha.3 consensus spec URL updates (#5065) 2023-06-13 14:03:49 +00:00
tersec 22208836b1
automated v1.4.0-alpha.1 to v1.4.0-alpha.2 consensus spec URL updates (#5056) 2023-06-10 09:56:54 +00:00
tersec 30c859e8a4
update some consensus-spec URLs to v1.4.0-alpha.1 (#5035) 2023-06-06 17:09:48 +00:00
Etan Kissling 12381d5f33
also pack attestations where LMD vote is orphaned (#5002)
* also pack attestations where LMD vote is orphaned

When `attestation.data.beacon_block_root` gets orphaned, attestations
with a good `attestation.data.target.root` may still be valuable.
The LMD GHOST vote is not relevant for attestation rewards.

Switch to use the FFG vote (`attestation.data.target.root`) instead,
gossip validation ensures it is an ancestor of `beacon_block_root`.

* lint
2023-06-04 10:39:12 +00:00
tersec f86febc111
update consensus spec URLs to v1.4.0-alpha.1 (#5027) 2023-06-04 10:32:50 +00:00
tersec b25ca0833b
use v1.4.0-alpha.1 consensus spec test vectors (#5026) 2023-06-03 21:55:08 +00:00
Etan Kissling d7890ac013
avoid port number conflict in `test_signing_node` (#5018)
Allow multiple copies of `test_signing_node` to run concurrently by
ensuring that they use unique network port numbers.
2023-06-01 09:33:25 +00:00
tersec 7a7573d7d0
fix chronos done -> completed deprecation warnings (#5016) 2023-06-01 10:04:30 +02:00
tersec c9f1bf21d6
refactor state diffs not to require two states in memory (#4986) 2023-05-30 11:55:32 +03:00
tersec 1f535336da
automated v1.3.0 to v1.4.0-alpha.0 consensus spec URL updates (#4996)
* automated v1.3.0 to v1.4.0-alpha.0 consensus spec URL updates

* add copyright year headers
2023-05-26 00:14:28 +02:00
tersec 10569ff2e9
use v1.4.0-alpha.0 consensus spec test vectors (#4990) 2023-05-25 07:55:00 +00:00
henridf 5ef748b19d
Clarify addOrphan error/logging (#4981)
* Clarify addOrphan error/logging

addOrphan returned a bool to indicate success. Change this to a Result
so that different errors can be distinguished.

* Update beacon_chain/consensus_object_pools/block_quarantine.nim

Co-authored-by: tersec <tersec@users.noreply.github.com>

* Update beacon_chain/gossip_processing/gossip_validation.nim

---------

Co-authored-by: tersec <tersec@users.noreply.github.com>
2023-05-21 17:47:00 +00:00
henridf 1cf777c64b
Fix sync for blocks older than MIN_EPOCHS_FOR_BLOB_SIDECARS_REQUESTS (#4977)
When doing sync for blocks older than
MIN_EPOCHS_FOR_BLOB_SIDECARS_REQUESTS, we skip the blobs by range
request, but we then pass en empty blob sequence to
validation, which then fails.

To fix this: Use an Option[Blobsidecars] to allow expressing the
distinction between "empty blob sequence" and "blobs unavailable". Use
the latter for "old" blocks, and don't attempt to run blob validation.
2023-05-19 16:25:11 +00:00
Zahary Karadjov a6b3e02ce0
Merge branch 'unstable' into dev/etan/rd-shufflingacc 2023-05-19 14:46:23 +03:00
Zahary Karadjov 4c3850f7df
Disable the use of incbin as it breaks the retail build on Linux/ARM 2023-05-19 04:08:02 +03:00
cheatfate a36cacda44
New stricter beacon-node URL parsing 2023-05-19 02:02:20 +03:00
Etan Kissling c70fd8fe97
Merge branch 'unstable' into dev/etan/rd-shufflingacc 2023-05-17 14:06:31 +02:00
tersec 74511f61d1
Use withdrawal credentials as default fee recipient (#4968) 2023-05-17 07:56:37 +03:00
Etan Kissling 40e89937c5
segregate sync committee messages by period / fork (#4953)
`SyncCommitteeMsgPool` grouped messages by their `beacon_block_root`.
This is problematic around sync committee period boundaries and forks.
Around sync committee period boundaries, members from both the current
and next sync committee may sign the same `beacon_block_root`; mixing
the signatures from both committees together is a mistake. Likewise,
around fork transitions, the `signing_root` changes, so those messages
also need to be segregated.
2023-05-17 07:55:55 +03:00
zah e8572c0246
More relaxed parsing of RemoteKeystores to regain compatibility with version 1 (#4967) 2023-05-16 19:06:16 +03:00
Jacek Sieka 83393cea8d
dependent slot helpers 2023-05-16 11:04:25 +02:00
Etan Kissling dbba003a38
Revert "Revert "accelerate `getShufflingRef` (#4911)" (#4958)"
This reverts commit 748be8b67b.
2023-05-15 17:41:40 +02:00
Etan Kissling 748be8b67b
Revert "accelerate `getShufflingRef` (#4911)" (#4958)
This reverts commit ea97e93e74.
2023-05-15 15:25:51 +00:00
henridf 573228ffa0
Rename eth1/ -> el/ and eth1_monitor.nim -> el_monitor.nim (#4944) 2023-05-15 05:05:12 +00:00
tersec 10a7281349
don't run flaky CI in Windows (#4935)
* don't run flaky CI in Windows

* reduce stack size in `test_light_client`

* regen tests

* re-enable maybe non-problematic test module on Windows

---------

Co-authored-by: Etan Kissling <etan@status.im>
2023-05-15 04:14:58 +02:00
Etan Kissling ec049bc493
use correct root in `test_gossip_validation` (#4951)
Use block root not state root for sync committee message in test.
2023-05-14 16:18:50 +02:00
Etan Kissling ea97e93e74
accelerate `getShufflingRef` (#4911)
When an uncached `ShufflingRef` is requested, we currently replay state
which can take several seconds. Acceleration is possible by:

1. Start from any state with locked-in `get_active_validator_indices`.
   Any blocks / slots applied to such a state can only affect that
   result for future epochs, so are viable for querying target epoch.
   `compute_activation_exit_epoch(state.slot.epoch) > target.epoch`

2. Determine highest common ancestor among `state` and `target.blck`.
   At the ancestor slot, same rules re `get_active_validator_indices`.
   `compute_activation_exit_epoch(ancestorSlot.epoch) > target.epoch`

3. We now have a `state` that shares history with `target.blck` up
   through a common ancestor slot. Any blocks / slots that the `state`
   contains, which are not part of the `target.blck` history, affect
   `get_active_validator_indices` at epochs _after_ `target.epoch`.

4. Select `state.randao_mixes[N]` that is closest to common ancestor.
   Either direction is fine (above / below ancestor).

5. From that RANDAO mix, mix in / out all RANDAO reveals from blocks
   in-between. This is just an XOR operation, so fully reversible.
   `mix = mix xor SHA256(blck.message.body.randao_reveal)`

6. Compute the attester dependent slot from `target.epoch`.
   `if epoch >= 2: (target.epoch - 1).start_slot - 1 else: GENESIS_SLOT`

7. Trace back from `target.blck` to the attester dependent slot.
   We now have the destination for which we want to obtain RANDAO.

8. Mix in all RANDAO reveals from blocks up through the `dependentBlck`.
   Same method, no special handling necessary for epoch transitions.

9. Combine `get_active_validator_indices` from `state` at `target.epoch`
   with the recovered RANDAO value at `dependentBlck` to obtain the
   requested shuffling, and construct the `ShufflingRef` without replay.

* more tests and simplify logic

* test with different number of deposits per branch

* Update beacon_chain/consensus_object_pools/blockchain_dag.nim

Co-authored-by: Jacek Sieka <jacek@status.im>

* `commonAncestor` tests

* lint

---------

Co-authored-by: Jacek Sieka <jacek@status.im>
2023-05-12 19:36:59 +02:00
Jacek Sieka b3c6320d56
embed genesis states using incbin (#4905) 2023-05-11 11:11:00 +00:00
Etan Kissling 22c298c07b
final portion of trivial v1.3.0 bumps (#4926)
* final portion of trivial v1.3.0 bumps

Updates unchanged logic to latest v1.3.0 consensus-specs refs.

```
https://github.com/ethereum/consensus-specs/(blob|tree)/(?!v1\.3\.0/)
```

* lint
2023-05-11 09:54:29 +00:00
Etan Kissling e44b51e955
fix `attachMerkleProofs` to support multiple deposits (#4932)
`attachMerkleProofs` is used by `mockUpdateStateForNewDeposit` to create
a single deposit. The function doesn't work correctly when trying with
with multiple deposits, though. Fix this to enable more complex tests,
and also return the `deposit_root` for forming matching `Eth1Data`.
2023-05-11 11:45:55 +03:00
tersec d3929cbb45
update some beacon API spec URLs; fix some Name and DuplicateModuleImport hints (#4929) 2023-05-10 10:20:55 +00:00
zah 5bf9284e62
Initial public version of the Verifying Web3Signer functionality (#4912)
* Allow the list of proved properties for web3signer to be configured
* Document the Web3Signer setups (regular, distributed and verified)
2023-05-09 11:16:43 +03:00
henridf 23adf15e5a
Blob handling sync fixes (#4888)
* Fix groupBlobs

* Fix getShortMap

* Fix blob handling in sync

* lint

* Add some blob-related logging
2023-05-06 08:58:50 +00:00
Etan Kissling 1ebcd8b473
`excess_data_gas` after `withdrawals` cleanup (#4900)
We already updated the field order in the actual `ExecutionPayload`,
but in init code and tests / logs etc we still used the old order.
Update those occurrences to also match the field order in the struct.
Furthermore, add `excess_data_gas` to last entry in `test_eth1_monitor`.
2023-05-05 23:15:47 +00:00
Etan Kissling 964417d8ba
cache computed mock pubkeys in tests (#4869)
We have a few tests that access `MockPubKeys`. Adding a cache provides
a minor speedup, as repeated pubkey computations are avoided.

64 cache hits: EF - Capella - Unittests - Light client - Sync protocol
64 cache hits: EF - Altair - Unittests - Light client - Sync protocol
256 cache hits: EF - Capella - Unittests - Light client - Sync protocol
256 cache hits: EF - Altair - Unittests - Light client - Sync protocol
2297 cache hits: Attestation pool processing [Preset: mainnet]
512 cache hits: Beacon chain DB [Preset: mainnet]
32 cache hits: Block processor [Preset: mainnet]
1082 cache hits: Validator change pool testing suite
192 cache hits: Gossip validation  [Preset: mainnet]
32 cache hits: Gossip validation - Extra
192 cache hits: Spec helpers
32 cache hits: Light client processor [Preset: mainnet]
96 cache hits: Light client [Preset: mainnet]
192 cache hits: Beacon state [Preset: mainnet]
37 cache hits: state diff tests [Preset: mainnet]
160 cache hits: Block pool processing [Preset: mainnet]
32 cache hits: Block pool altair processing [Preset: mainnet]
96 cache hits: chain DAG finalization tests [Preset: mainnet]
32 cache hits: Old database versions [Preset: mainnet]
64 cache hits: Diverging hardforks
96 cache hits: Backfill
32 cache hits: Starting states
32 cache hits: Latest valid hash [Preset: mainnet]
32 cache hits: Pruning

5912 cache hits total per `make -j test`.
2023-04-27 15:04:56 +00:00