Commit Graph

3236 Commits

Author SHA1 Message Date
tersec 629b005c27
refactor batch validation not to require genesis_validators_root each time (#4640) 2023-02-20 09:26:22 +01:00
tersec 0cc0c7e6b7
builder spec v0.3.0 URL updates (#4641)
* builder spec v0.3.0 URL updates

* remove stray non-URL/comment change
2023-02-18 01:54:30 +01:00
tersec a382498cfe
batch-verify BLS to execution change messages (#4637) 2023-02-17 13:35:12 +00:00
tersec dc0bbe3a57
rm blockForkAtEpoch and switch callers to consensusForkAtEpoch (#4634) 2023-02-16 21:16:54 +01:00
Zahary Karadjov 8202d4a84e
Version 23.2.0 2023-02-16 20:19:22 +02:00
Eugene Kabanov e91415662b
Keystore cache implementation. (#4372) 2023-02-16 19:25:48 +02:00
Jacek Sieka d63179ab57
vc: avoid npe for unknown validator (#4632)
follow-up for #4590
2023-02-16 13:27:10 +00:00
tersec e342fdd97a
rename stateForkAtEpoch to consensusForkAtEpoch (#4627) 2023-02-16 11:32:12 +02:00
zah ff464e49cf
Implement the set of gas_limit end-points in the Keymanager API (#4612)
Fixes #3946
2023-02-15 15:10:31 +00:00
zah 067ba13c52
Capella metadata for Sepolia (#4615)
Other changes:

Renamed the `EIP_4844_FORK_*` config constants to `DENEB_FORK_*` as
this matches the latest spec and it's already used in the official
Sepolia config.
2023-02-15 14:44:09 +00:00
tersec b7c3037f01
consensus spec ref URL updates to v1.3.0-rc.2 (#4626) 2023-02-15 14:43:51 +00:00
Eugene Kabanov 218ea42220
VC: Fix getBlockRoot() response with execution_optimistic. (#4622)
* Fix getStateRoot() and getBlockRoot() API functions which should obtain `execution_optimistic` field.
Fix sync committee service to check `execution_optimistic` field of getBlockRoot() response.

* 2nd part.

* Remove presets usage.
2023-02-15 15:09:31 +01:00
Etan Kissling 1ac7f1a47a
rm unused `lowParticipation` flag (#4624)
Remove an unused flag and associated (unreachable) logic, as the flag
wasn't even passed to that module.
2023-02-15 00:25:22 +01:00
tersec cf551f10c4
don't fcU on blocks for which block processor received no newPayload reply (#4623) 2023-02-14 21:41:49 +01:00
Etan Kissling 32f6309f95
pass correct CI test flag to initializer (#4621)
Use correct `experimental` flag for running tests.
2023-02-14 14:51:24 +01:00
tersec 3977f1529a
fill in remaining capellaImplementationMissing holes for builder API (#4606) 2023-02-14 11:49:48 +01:00
tersec 3011d49946
refactor fcU sending and rename EL-side root to hash (#4614) 2023-02-14 07:48:39 +01:00
tersec 07ccd3fa6e
remove capella and deneb empty execution payload fallbacks (#4613) 2023-02-13 19:15:16 +02:00
tersec aee19fec6b
block on forkchoiceUpdated EL calls due to doing fewer of them (#4609) 2023-02-13 12:13:52 +01:00
tersec d838e5b6ba
remove unused phase0/altair-specific REST deserialization (#4605) 2023-02-13 12:03:38 +01:00
tersec f9022618f0
deprecate req/resp GetMetaData v1 (#4608) 2023-02-13 12:02:20 +01:00
henridf 59e41dc65d
EIP4844 sync (#4581)
* EIP4844 Sync

* Pass eip4844 fork epoch rather than cfg to syncmanager

* Fix sync

* Update test

* map->mapIt
2023-02-11 20:48:35 +00:00
Etan Kissling 09dd64df32
add deployment phase for Capella-ready networks (#4607)
Allow distinguishing Capella-ready networks from non-upgraded networks
based on `CAPELLA_FORK_EPOCH` being set (Zhejiang public testnet).
2023-02-10 14:26:01 +01:00
tersec 20fc355ff1
final batch of automated 1.3.0-rc.2 consensus-spec urls (#4604) 2023-02-09 22:08:43 +00:00
Jacek Sieka 856fcea8d7
fix slow checking of unknown validators (#4590)
We do a linear scan of all pubkeys for each validator and slot - this
becomes expensive with large validator counts.

* normalise BN/VC validator startup logging
* fix crash when host cannot be resolved while adding remote validator
* silence repeated log spam for unknown validators
* print pubkey/index/activation mapping on startup/validator
identification
2023-02-07 14:53:36 +00:00
Jacek Sieka 98db005c70
seed sync committee state cache (#4592)
By pre-seeding the sync committee cache when applying blocks, we avoid a
significantly expensive validator set traversal / sync committee index
construction during sync / block application - 20-30% sync speedup
post-altair.

* also cache/reload total active balance for another cool 10%
2023-02-07 11:22:22 +01:00
tersec ebb45e10da
initial implementation of capella MEV (#4567) 2023-02-06 19:07:30 +01:00
Jacek Sieka f3ddea6c86
Skip execution payload verification for finalized blocks (#4591)
While syncing the finalized portion of the chain, the execution client
cannot efficiently sync and most of the time returns `SYNCING` - in this
PR, we use CL-verified optmistic sync as long as the block is claimed to
be finalized, only occasionally updating the EL with progress.

Although a peer might lie about what is finalized and what isn't,
eventually we'll call the execution client - thus, all a dishonest
client can do is delay execution verification slightly. Gossip blocks in
particular are never assumed to be finalized.
2023-02-06 08:22:08 +01:00
henridf 9233a52e4b
Don't pass RuntimeConfig to RequestManager (#4589) 2023-02-04 12:13:09 +01:00
tersec bca781b1b0
prioritize REST API-provided BLS to execution changes over gossip-received changes (#4580) 2023-02-03 16:28:28 +01:00
tersec 1c62a5eb24
capella VC support (#4586) 2023-02-03 16:12:11 +01:00
tersec 956aee2d35
fill some capella/EIP4844 missing implementations (#4585) 2023-02-02 22:24:06 +00:00
tersec eb56762d41
some proc -> func in signatures_batch (#4582) 2023-02-02 19:05:30 +00:00
tersec 63ed5885ab
update engine API URLs to v1.0.0-beta.2 (#4579) 2023-02-01 18:49:36 +00:00
tersec bcc9781cc7
rm obsolete interop module (#4570) 2023-02-01 16:29:55 +01:00
henridf 94837caa2a
Eip4844 sync fixes (#4577)
* fixes

* Make some log messages blob-aware

* remove redundant optBlobs()
2023-02-01 14:14:50 +00:00
henridf d7316ac863
Wire up blob fetching with request manager (#4527)
* Wire up blob fetching and request manager

* Update beacon_chain/sync/request_manager.nim

Co-authored-by: tersec <tersec@users.noreply.github.com>

* Review feedback: remove redundant RequestManager field.

* Fix syntax

* Improve blob fetching logic

* fix previous commit

* rman.isBlobsTime(): Remove spurious parameter

* Review feedback

* expressionify an if

---------

Co-authored-by: tersec <tersec@users.noreply.github.com>
2023-02-01 00:25:08 +01:00
tersec cdf69b9360
more automated v1.3.0-rc.1 to v1.3.0-rc.2 consensus spec URL updates (#4572) 2023-01-31 22:49:17 +00:00
Etan Kissling ea6a6b1acd
track slot as part of fork choice debug API (#4565)
Extends fork choice state to also track slot numbers to improve accuracy
of `/eth/v1/debug/fork_choice` endpoint. Autoenable this API on devnet,
and disable some extra checks on devnet to aid focused testing efforts.
Align fork choice pruning logic with API based on checkpoints vs root.
2023-01-31 12:35:01 +00:00
tersec 58ed9308d2
automated v1.3.0-rc.1 to v1.3.0-rc.2 consensus spec URL updates (#4568) 2023-01-31 00:26:57 +01:00
tersec f5e302f973
use 1.3.0-rc.2-hotfix consensus spec test vectors (#4566) 2023-01-30 22:54:16 +00:00
tersec b2f415cbe5
remove merge cruft (#4559) 2023-01-30 20:22:05 +01:00
tersec 8e2792bdd7
remove Nim 1.2 workarounds and `TODO`s (#4533)
* clean up some Nim 1.2 workarounds

* re-add notes about JS backend

* another proc/noSideEffect -> func

* revert ncli/ncli_common.nim changes; 19969 evidently wasn't backported to 1.6
2023-01-30 20:21:51 +01:00
tersec 29ea42ce1a
use v1.3.0-rc.2 consensus spec test vectors (#4564) 2023-01-29 11:28:04 +00:00
tersec 0fb726c420
`BeaconStateFork/BeaconBlockFork` -> `ConsensusFork` (#4560)
* `BeaconStateFork/BeaconBlockFork` -> `ConsensusFork`

* revert unrelated change

* revert unrelated changes

* update test summaries
2023-01-28 19:53:41 +00:00
henridf 67e2f854d5
Fix BeaconBlockAndBlobsSidecar encoding (#4562) 2023-01-28 12:30:38 +00:00
tersec 7635b47a4e
update/clarify Nim issue link after retesting/re-isolating 2023-01-27 10:41:19 +00:00
Etan Kissling 9f279e2933
cache LC headers and sync committees for bootstrap (#4499)
To allow LC data retention longer than the one for historic states,
introduce persistent DB caches for `current_sync_committee` and
`LightClientHeader` for finalized epoch boundary blocks.
This way, historic `LightClientBootstrap` requests may still be honored
even after pruning. Note that historic `LightClientUpdate` requests are
already answered using fully persisted objects, so don't need changes.

Sync committees and headers are cached on finalization of new data.
For existing data, info is lazily cached on first access.

Co-authored-by: Jacek Sieka <jacek@status.im>
2023-01-27 10:44:57 +01:00
Zahary Karadjov daa99f43c3
Merge branch 'stable' into unstable 2023-01-26 16:47:26 +01:00
Eugene Kabanov 08ed8ad43e
Adopt BN and VC header sizes and requirements, to avoid users confusion with default configuration options. (#4556)
Add comments.
2023-01-26 16:00:10 +01:00
tersec c22384d7ba
always use Bellatrix message size limits (#4558) 2023-01-26 14:24:23 +00:00
Zahary Karadjov 1ea102b033
Version 23.1.1 2023-01-26 14:37:03 +01:00
Jacek Sieka 0c1686eb20
fix pruned backfill summary (fixes #4552)
When enabling pruned history mode, the backfill summary as it was
computed with full backfilling may be pruned - don't try to load it on
init
2023-01-26 14:23:12 +01:00
Jacek Sieka fa27ce76a3
vmon: set details mode to default off
The "on" default for validator monitor details incurs a heavy
performance penalty on large-validator setups - this may cause excess
memory usage or slowdowns when metrics are queried - this PR changes the
default to off, as was intended for the 23.1.0 release.
2023-01-26 14:18:52 +01:00
Jacek Sieka 968e27bc97
fix pruned backfill summary (fixes #4552) (#4554)
When enabling pruned history mode, the backfill summary as it was
computed with full backfilling may be pruned - don't try to load it on
init
2023-01-26 11:23:17 +01:00
henridf 03b468c537
Pass correct block root to validate_blobs_sidecar (#4557) 2023-01-26 09:25:37 +00:00
henridf 7966ab6be2
Some EIP4844 fixes (#4549)
* debug log upon sidecar validation failure

* Fill in signature catch upon SignedBeaconBlockAndBlobsSidecar deser

* Always fill blobssidecar slot and root

* Skip lastFCU when eth1monitor is nil

* fix

* Use cached root
2023-01-25 18:35:46 +01:00
henridf d349a4797f
Fix minimum_request_epoch in BeaconBlockAndBlobsSidecarByRoot (#4555) 2023-01-25 18:33:01 +01:00
tersec d30544f790
disallow filling BLS to execution change pool pre-capella (#4547)
* disallow filling BLS to execution change pool pre-capella

* use better BLS change epoch
2023-01-24 22:25:51 +01:00
Etan Kissling efbd4e395a
avoid sending redundant LC finality updates (#4546)
When the epoch boundary block is missed, we incorrectly assume that the
next couple blocks improve finality, leading to repeated pushes of the
same light client finality update and incorrectly ignoring some gossip.
2023-01-24 17:44:55 +00:00
tersec fe1a57c220
use shortLog for execution payload logging (#4544) 2023-01-24 13:19:38 +00:00
Jacek Sieka ba0496105a
ncli_db: fix readonly properly (#4543)
🐡

Co-authored-by: Etan Kissling <etan@status.im>
2023-01-24 02:52:08 +00:00
Etan Kissling 0b8bb11c50
combine common implementation of LC helpers (#4542)
* combine common implementation of LC helpers

Combine replicated helper code from Altair/Capella/EIP4844 into single
`Forky` based implementation. Also convert `template` to `func` to avoid
selection of incorrect overload.

* fix
2023-01-23 19:38:03 +00:00
Zahary Karadjov 285eec6512
Add metrics and debug logging for dropped BLS to execution change messages 2023-01-23 14:58:40 +01:00
henridf 349001b7fb
eip4844 beacon block proposals (#4540)
* eip4844 beacon block proposals

* Don't fetch blobs under minimal preset

@tersec's summary of the issue:

BlobsBundleV1 in the execution API spec assumes a mainnet preset blob
size, where the EIP4844 consensus spec defines
FIELD_ELEMENTS_PER_BLOB: 4 under the minimal preset, which leads to a
Blob having a length of 4 * 32, not 4096 * 32 which BlobsBundleV1
requires.

* Revert unintentional script change
2023-01-21 23:13:21 +00:00
tersec 0a93023f46
capella fcU usage updating duly ensured (#4539) 2023-01-21 13:45:23 +00:00
henridf 96dfe18a3d
make message_router.routeSignedBeaconBlock blob-aware (#4537)
* make message_router.outeSignedBeaconBlock blob-aware

* Update beacon_chain/spec/forks.nim

* Update beacon_chain/spec/forks.nim
2023-01-21 06:34:04 +00:00
Etan Kissling a57cec56cb
update EIP4844 related light client sync docs (#4538)
Further clarify why it is okay to drop blobs during LC based sync.
2023-01-21 06:23:28 +00:00
henridf f8ee0def2b
Add stubs for EIP4844 engine API calls (#4536) 2023-01-21 00:47:38 +00:00
tersec 8c2e01da24
consensus spec ref URL updates to v1.3.0-rc.1 (#4534) 2023-01-20 22:33:05 +00:00
tersec 6a64048e80
remove vestigial/unused BLSToExecutionChange parameters (#4535) 2023-01-20 20:51:54 +00:00
tersec aacc8d702d
remove Nim 1.2-compatible `push raise`s and update copyright notice years (#4528) 2023-01-20 14:14:37 +00:00
Etan Kissling a5596dc5e6
add documentation for `shouldSyncOptimistically` (#4530)
Clarify rationale behind the `shouldSyncOptimistically` workaround for
why we subscribe to blocks gossip when light client syncing.
2023-01-20 05:31:05 +01:00
tersec cd58613615
remaining automated rc.0 to rc.1 consensus spec ref URL changes (#4529) 2023-01-20 00:20:51 +00:00
Etan Kissling 65ca523482
support EIP4844 LC data format (#4520)
Implements the proposed light client data format for EIP4844:
- https://github.com/ethereum/consensus-specs/pull/3151
2023-01-19 23:53:37 +01:00
tersec 819e007689
exit/validatorchange pool includes BLS to execution messages; REST support for new pool (#4519)
* exit/validatorchange pool includes BLS to execution messages; REST
support for new pool

* catch failed individual futures

* increase BLS changes bound and keep BLS seen consistent with subpool

* deque capacities should be powers of 2
2023-01-19 22:00:40 +00:00
Zahary Karadjov ba7c0bc091
Version 23.1.0 2023-01-18 19:42:41 +02:00
zah ae301af291
Fix #4500 (#4523) 2023-01-18 19:20:28 +02:00
Etan Kissling 58f4c13e14
support Capella LC data format (#4514)
Implements the proposed light client data format for Capella:
- https://github.com/ethereum/consensus-specs/pull/3151
2023-01-18 15:32:57 +01:00
tersec aea7a0c8b8
remove TTD monitoring (#4486) 2023-01-18 16:01:49 +02:00
Etan Kissling 9cceb1b4a0
support `readOnly` backend in LC client/data dbs (#4518)
When accessing DB in `readOnly` mode that does not already have latest
schema, initial writes trigger `attempt to write a readonly database`.
Avoid that by only writing schema when DB is not `readOnly`, and provide
data from legacy tables if such are present.
2023-01-18 03:06:23 +01:00
tersec 073c544f0c
automated update from v1.3.0-rc.0 to v1.3.0-rc.1 consensus spec URLs (#4517) 2023-01-17 16:10:52 +00:00
tersec 9b0868033e
use GitHub Actions cache and upload-artifact v3 (#4516) 2023-01-17 15:45:04 +00:00
henridf 05a6c6280d
Add `beacon_block_and_blobs_sidecar_by_root` RPC (#4475)
* Implement beacon_block_and_blobs_sidecar_by_root

* review feedback

* more review feedback
2023-01-16 17:08:16 +00:00
henridf 727920a571
Refactor block/blobs types (#4491)
* Refactor block/blobs types

Use type system to enforce invariant that a pre-4844 block cannot have
a sidecar.

* Update beacon_chain/nimbus_beacon_node.nim

Co-authored-by: tersec <tersec@users.noreply.github.com>

* review feedback

Co-authored-by: tersec <tersec@users.noreply.github.com>
2023-01-16 16:26:48 +00:00
Etan Kissling fda03548e3
use `ForkedLightClientStore` internally (#4512)
When running `nimbus_light_client`, we persist the latest header from
`LightClientStore.finalized_header` in a database across restarts.
Because the data format is derived from the latest `LightClientStore`,
this could lead to data being persisted in pre-release formats.

To enable us to test later `LightClientStore` versions on devnets,
transition to a `ForkedLightClientStore` internally that is only
migrated to newer forks on-demand (instead of starting at latest).
2023-01-16 16:53:45 +01:00
Jacek Sieka d8caab500d
Enable validator monitor by default (#4468)
By enabling the validator monitor, more precise information about the
lifecycle of an attestation is logged at the higher `NOTICE` log level
while current `sent` messages are logged at `INF` instead, since they
are less interesting.

In particular, missed attestations and those that vote for the wrong
head are now detected and logged at NOTICE.

In addition to logging, this feature enables rich metrics around
attestation and sync committee performance - by default, validators are
tracked in aggregate but a detailed mode exists as well

This feature has been available since early Nimbus days, but it has now
been tuned and optimised such that it is safe to enable by default, even
for large setups.

* enable automatic validator monitoring by default
* replace `--validator-monitor-totals` flag with
`--validator-monitor-details` - the detailed mode is disabled by default
* lower "sent" log level to `INF` for several messages - in particular
those that are traced by the validator monitor

This is a retake on #3531 which was later reverted in #3578.
2023-01-16 11:28:35 +01:00
Etan Kissling 609227559f
LC data fork cleanup (#4506)
Distinguish between those code locations that need to be updated on each
light client data format change, and those others that should generally
be fine, as long as a valid light client object is processed.

The former are tagged with static assert for `LightClientDataFork.high`.

The latter are changed to `lcDataFork > LightClientDataFork.None` to
indicate that they depend only on presence of any valid object.
Also bundled a few minor cleanups and fixes.

Also add `Forky` type for `LightClientStore` and minor fixes / cleanups.
2023-01-14 22:19:50 +01:00
tersec eb02415f00
execution/engine withdrawals amount in uint64 gwei (#4509) 2023-01-14 17:26:57 +00:00
tersec 68e08c6f15
use v1.3.0-rc.1 consensus spec test vectors (#4505) 2023-01-14 04:20:51 +00:00
Etan Kissling 2324136552
add `LightClientHeader` wrapper (#4481)
The light client data structures were changed to accommodate additional
fields in future forks (e.g., to also hold execution data).

There is a minor change to the JSON serialization, where the `header`
properties are now nested inside a `LightClientHeader`.
The SSZ serialization remains compatible.

See https://github.com/ethereum/consensus-specs/pull/3190
and https://github.com/ethereum/beacon-APIs/pull/287
2023-01-13 16:46:35 +01:00
henridf 77530841e3
Fix eip4844 gossip transition (#4498)
* Fix eip4844 gossip transition

* Future-proof block topic selection

* whitespace
2023-01-13 15:37:31 +00:00
Zahary Karadjov 46fc5716a4
Faster keystore generation in the local sim 2023-01-13 12:31:39 +02:00
zah 0f758c5f02
Working Makefile targets for Capella devnet2 (#4494)
* Working Makefile targets for Capella devnet2

make capella-devnet-2
make clean-capella-devnet-2

You'll need to have https://github.com/tmuxinator/tmuxinator installed.
It's available as a regular package in most Linux distributions or through
Nix or Brew on macOS.

This commit also fixes the initial hang in the Eth1 monitor in the "find
TTD block" procedure through a fix to the network metadata files which
hasn't been upstreamed yet.

Other changes:

* Disabled Geth snap sync in the simulation

When all Geth nodes are configured to run with snap sync enabled, they all
start snap sync after the first forkchoiceUpdated which causes the BNs to
skip validator duties because the EL is syncing. The snap sync never completes
due to poor connectivity between the Geth nodes in the simulation.
2023-01-13 12:21:58 +02:00
Zahary Karadjov 3b03ef8ffb
Remove the genesis detection code 2023-01-13 04:28:30 +02:00
Zahary Karadjov b06502bf65
Gnosis const preset 2023-01-13 04:28:29 +02:00
Zahary Karadjov c01e53e35e
Fix bitrot in the gnosis build; Up-to-date bootstrap nodes
To prevent similar bitrot from occuring in the future, the gnosis-build
is now part of the default `make` targets.
2023-01-13 04:28:29 +02:00
henridf 9da64bb605
Far-future epoch check before installing 4844 vals (#4497) 2023-01-12 21:46:12 +01:00
Etan Kissling 7e276937dc
make LC data fork aware (#4493)
In a future fork, light client data will be extended with execution info
to support more use cases. To anticipate such an upgrade, introduce
`Forky` and `Forked` types, and ready the database schema.
Because the mapping of sync committee periods to fork versions is not
necessarily unique (fork schedule not in sync with period boundaries),
an additional column is added to `period` -> `LightClientUpdate` table.
2023-01-12 18:11:38 +01:00
Jacek Sieka 6bfc766629
drop subset sync contributions in gossip (#4490)
* correctly report ignored contributions in metrics
* avoid counting subset contributions in vmon (bring in line with
attestation aggregates)
* avoid signature checks for subset attestations

A being a non-strict subset is a sufficient condition to ignore.
2023-01-12 15:08:08 +01:00
tersec 4f9064a153
consensus spec ref URL updates (#4496) 2023-01-12 13:33:14 +00:00
henridf 5a03e7400e
Blobs sidecars by range (#4464)
* Add blobs_sidecars_by_range RPC

* Update beacon_chain/spec/datatypes/eip4844.nim

Co-authored-by: tersec <tersec@users.noreply.github.com>

Co-authored-by: tersec <tersec@users.noreply.github.com>
2023-01-12 12:33:24 +00:00
henridf 309f8690de
Wire up engine_newPayloadV3 (#4482)
* Wire up eip4844's newPayloadV3

* Add eip4844 test

* Update AllTests-mainnet.md and fix typo
2023-01-11 18:21:19 +00:00
tersec b14106b72b
preserve historical_summaries in capella to eip4844 fork (#4492) 2023-01-11 17:21:16 +00:00
Jacek Sieka ba3db7aa5a
spec: Option -> Opt (#4488) 2023-01-11 12:29:21 +00:00
Zahary Karadjov 713bdd317d
Bugfix: Handle networks with TTD=0 (e.g. withdrawal devnets) 2023-01-10 19:17:11 +02:00
tersec e28e1aeec8
a few consensus spec ref URL updates (#4483) 2023-01-10 16:14:17 +00:00
Etan Kissling ca01356d8e
stricter check for context bytes when recv LC data (#4480)
Bellatrix and Altair light client data share same body, but have other
fork digests. Validate that the peer's sent fork digest matches the one
that we expect (derived from `attested_header.slot`).
2023-01-10 16:54:29 +01:00
tersec 2fc0fa4288
don't use non-engine-API-whitelisted web3 methods (#4484) 2023-01-10 15:50:34 +00:00
Etan Kissling 0590be7afe
add `num_active_participants` helpers for `SyncAggregate` (#4478)
Introduce `num_active_participants` helper function to reduce visibility
of low-level `countOnes` function and reduce code duplication.
2023-01-10 11:26:25 +00:00
Etan Kissling 57b04c9f9b
disable light client gossip from Capella onward (#4479)
To prevent exchanging incompatible data and therefore disconnect,
disable light client data gossip from Capella onward until specs are ok:
https://github.com/ethereum/consensus-specs/pull/3151
2023-01-09 23:49:56 +00:00
tersec 2dd3cd786f
consensus spec ref URL update v1.3.0-{alpha.2,rc.0}; copyright year update (#4477) 2023-01-09 22:44:44 +00:00
henridf 64878888bd
Blob storage (#4454)
* Blob storage

* fix indentation

* Fix build (none->Opt.none)

* putBlobs -> putBlobsSidecar

* getBlobs -> getBlobsSidecar

* Check blob correctness when storing a backfill block

* Blobs table: rename and conditionally create

* Check block<->blob match in storeBackfillBlock

* Use when .. toFork() to condition on type

* Check blob viability in block_processor.storeBlock()

* Fix build

* Review feedback
2023-01-09 18:42:10 +00:00
tersec 787703bd0c
use v1.3.0-rc.0 consensus spec test vectors (#4472) 2023-01-09 15:15:43 +01:00
Jacek Sieka 0ba9fc4ede
History pruning (fixes #4419) (#4445)
Introduce (optional) pruning of historical data - a pruned node will
continue to answer queries for historical data up to
`MIN_EPOCHS_FOR_BLOCK_REQUESTS` epochs, or roughly 5 months, capping
typical database usage at around 60-70gb.

To enable pruning, add `--history=prune` to the command line - on the
first start, old data will be cleared (which may take a while) - after
that, data is pruned continuously.

When pruning an existing database, the database will not shrink -
instead, the freed space is recycled as the node continues to run - to
free up space, perform a trusted node sync with a fresh database.

When switching on archive mode in a pruned node, history is retained
from that point onwards.

History pruning is scheduled to be enabled by default in a future
release.

In this PR, `minimal` mode from #4419 is not implemented meaning
retention periods for states and blocks are always the same - depending
on user demand, a future PR may implement `minimal` as well.
2023-01-07 10:02:15 +00:00
tersec d1b799eb64
bump nim-web3 for correct getPayloadV2 response signature (#4471) 2023-01-06 21:06:45 +00:00
tersec 47cb0f7991
capella forkchoiceUpdated support (#4462)
* capella forkchoiceUpdated support

* match V2 fcU with V2 getPayload
2023-01-06 22:01:10 +01:00
tersec c5d1683f19
spec ref URL & copyright year updates (#4467) 2023-01-06 16:28:46 +00:00
henridf 2fc08dfb3f
EIP4844 gossip transition (#4463) 2023-01-05 21:35:07 +00:00
henridf 3a84d61b2e
db.getBlockSZ: Remove no-op 'success' var (#4461) 2023-01-05 17:31:46 +00:00
tersec ec01065555
getPayloadV2 support for capella (#4457)
* getPayloadV2 support for capella

* check execution payload type more rigorously
2023-01-04 20:13:17 +01:00
Jacek Sieka 7c2ed5c609
Always-on optimistic mode (#4458)
With https://github.com/status-im/nimbus-eth2/pull/4420 implemented, the
checks that we perform are equivalent to those of a `SYNCING` EL - as
such, we can treat missing EL the same as SYNCING and proceed with an
optimistic sync.

This mode of operation significantly speeds up recovery after an offline
EL event because the CL is already synced and can immediately inform the
EL of the latest head.

It also allows using a beacon node for consensus archival queries
without an execution client.

* deprecate `--optimistic` flag
* log block details on EL error, soften log level because we can now
continue to operate
* `UnviableFork` -> `Invalid` when block hash verification fails -
failed hash verification is not a fork-related block issue
2023-01-04 15:51:14 +00:00
henridf 8251cc223d
eip4844 gossip (#4444)
* eip4844 gossip

* Check BLSFieldElement range validity in gossip validation

* lint/nits cleanup

* Use template to avoid an assignment with copy.

* More review feedback

* lint

* lint

* processSignedBeaconBlockAndBlobsSidecar: clean up error handling flow

* Undo factoring-out of beacon blocks validator installation
2023-01-04 12:34:15 +00:00
Jacek Sieka aff0505807
Allow reindexing to start from an arbitrary state (#4437)
When not backfilling all the way to genesis (#4421), it becomes more
useful to start rebuilding the historical indices from an arbitrary
starting point.

To rebuild the index from non-genesis, a state and an unbroken block
history is needed - here, we allow loading the state from an era file
and recreating the history from there onwards.

* speed up partial era state loading
2023-01-03 20:37:09 +01:00
tersec 45654984a9
capella validator withdrawal credentials aren't immutable (#4455) 2023-01-03 20:04:59 +01:00
tersec 121c66ad3e
fix getForkedBlock to read EIP4844 and Capella blocks (#4450)
* fix getForkedBlock to read EIP4844 and Capella blocks

* update geth develop URLs
2022-12-30 21:59:21 +00:00
Jacek Sieka d854317646
fix countdown over time period (#4449)
*sigh*

fixes:
```
Error: unhandled exception:
beacon_chain/beacon_chain_db_light_client.nim(214, 12)
`period.isSupportedBySQLite`  [AssertionDefect]
```
2022-12-30 21:36:31 +01:00
Jacek Sieka 75c7195bfd
Backfill only up to MIN_EPOCHS_FOR_BLOCK_REQUESTS blocks (#4421)
When backfilling, we only need to download blocks that are newer than
MIN_EPOCHS_FOR_BLOCK_REQUESTS - the rest cannot reliably be fetched from
the network and does not have to be provided to others.

This change affects only trusted-node-synced clients - genesis sync
continues to work as before (because it needs to construct a state by
building it from genesis).

Those wishing to complete a backfill should do so with era files
instead.
2022-12-23 08:42:55 +01:00
Etan Kissling 2ac7609259
display owl on Capella transition (#4442)
Trigger ANSI art on upgrade to Capella, similar to the merge.
Future extension could log blinking art when user successfully managed
to get BLS to Execution change included into a block for a validator.

Art created by http://beatscribe.com/ (beatscribe#1008 on Discord)
2022-12-21 13:30:24 +01:00
Etan Kissling 12b428e3d7
fix `produceBlockV2` call for Capella (#4443)
Avoid issue when calling `/eth/v2/validator/blocks/{slot}` for a
Capella slot via VC, same logic as in `validator_duties`.
2022-12-21 10:35:56 +00:00
Zahary Karadjov adcc57e7ca
Version 22.12.0 2022-12-21 10:50:13 +02:00
Etan Kissling 2184fd2322
bump `nim-eth`, extend empty block fallback for EIP4844 (#4425)
Implements `emptyPayloadToBlockHeader` for EIP-4844.
https://github.com/status-im/nim-eth/pull/570
2022-12-20 20:00:56 +01:00
Etan Kissling c91d9d61e2
validate EL block hash in EL simulation (#4420)
When simulating EL with `--optimistic` flag, perform block hash check.
2022-12-20 09:24:33 +01:00
zah 07d4160e00
Migrating the deposit contract snapshot can no longer fail on start-up (#4438)
The missing piece of data that had to be obtained previously from
the configured EL client is now part of the network metadata baked
into the binary.
2022-12-19 18:19:48 +01:00
Jacek Sieka bd8f08204e
Implement skip_randao_verification for blinded blocks (#4435)
* Implement skip_randao_verification for blinded blocks

* fix redundant randao verification on block replay (5% faster)
* check randao in REST instead of internally
* avoid redundant copies when making blocks
* cleanup leftover randao skipping code

* fix test summary
2022-12-19 15:11:12 +02:00
Jacek Sieka 7501f10587
60% state replay speedup (#4434)
* 60% state replay speedup

* don't use HashList for epoch participation - in addition to the code
currently clearing the caches several times redundantly, clearing has to
be done each block nullifying the benefit (35%)
* introduce active balance cache - computing it is slow due to cache
unfriendliness in the random access pattern and bounds checking and we
do it for every block - this cache follows the same update pattern as
the active validator index cache (20%)
* avoid recomputing base reward several times per attestation (5%)

Applying 1024 blocks goes from 20s to ~8s on my laptop - these kinds of
requests happen on historical REST queries but also whenever there's a
reorg.

* fix test and diffs
2022-12-19 14:01:49 +02:00
Jacek Sieka 064d164a88
fix capella+ summary loading (#4433)
...once and for all.
2022-12-16 13:11:08 +01:00
tersec 110401394b
consolidate consensus spec transition test fixtures (#4428)
* consolidate consensus spec transition test fixtures

* include capella

* consoliate fork test fixtures

* note change in EIP-4844 process_block in alpha.2
2022-12-15 22:51:04 +00:00
tersec bb4ea37baa
update EF consensus spec URLs from v1.3.0-alpha.1 to v1.3.0-alpha.2 (#4432) 2022-12-15 12:15:12 +00:00
tersec d269d82b71
use v1.3.0-alpha.2 EF consensus spec test vectors (#4430) 2022-12-15 01:51:06 +00:00
tersec e7706768c3
add database beaconstate tests for capella and eip4844 (#4429) 2022-12-14 23:12:29 +00:00
tersec 7faef7827e
fix EIP4844 withBlck (#4411)
* fix EIP4844 withBlck

* don't raiseAssert by default
2022-12-14 18:30:56 +01:00
Jacek Sieka 2440954555
allow syncing capella blocks (#4423) 2022-12-14 01:30:34 +00:00
tersec cd993ca418
consolidate slot sanity checks across forks (#4418)
* consolidate slot sanity checks across forks

* use `new` rather than `init` for constructor of `ref`s
2022-12-13 15:58:59 +00:00
tersec bc996623e0
add EIP4844 block database read/write test (#4416) 2022-12-13 00:56:50 +00:00
Zahary Karadjov 478e6f7c38
merge stable into unstable 2022-12-13 01:24:12 +02:00
Zahary Karadjov e4b1933773
Version 22.11.1 2022-12-12 20:18:57 +02:00
tersec 6a80b56d82
use VC-provided graffiti for blinded block proposals (#4417) 2022-12-12 20:05:54 +02:00
tersec 6057631741
fix bellatrix web3signer client-side beacon block signing requests (#4407) 2022-12-12 20:04:20 +02:00
tersec 45574021c3
use VC-provided graffiti for blinded block proposals (#4417) 2022-12-12 13:02:06 +00:00