Commit Graph

379 Commits

Author SHA1 Message Date
tersec 8e2792bdd7
remove Nim 1.2 workarounds and `TODO`s (#4533)
* clean up some Nim 1.2 workarounds

* re-add notes about JS backend

* another proc/noSideEffect -> func

* revert ncli/ncli_common.nim changes; 19969 evidently wasn't backported to 1.6
2023-01-30 20:21:51 +01:00
tersec 0fb726c420
`BeaconStateFork/BeaconBlockFork` -> `ConsensusFork` (#4560)
* `BeaconStateFork/BeaconBlockFork` -> `ConsensusFork`

* revert unrelated change

* revert unrelated changes

* update test summaries
2023-01-28 19:53:41 +00:00
Etan Kissling 9f279e2933
cache LC headers and sync committees for bootstrap (#4499)
To allow LC data retention longer than the one for historic states,
introduce persistent DB caches for `current_sync_committee` and
`LightClientHeader` for finalized epoch boundary blocks.
This way, historic `LightClientBootstrap` requests may still be honored
even after pruning. Note that historic `LightClientUpdate` requests are
already answered using fully persisted objects, so don't need changes.

Sync committees and headers are cached on finalization of new data.
For existing data, info is lazily cached on first access.

Co-authored-by: Jacek Sieka <jacek@status.im>
2023-01-27 10:44:57 +01:00
Jacek Sieka 968e27bc97
fix pruned backfill summary (fixes #4552) (#4554)
When enabling pruned history mode, the backfill summary as it was
computed with full backfilling may be pruned - don't try to load it on
init
2023-01-26 11:23:17 +01:00
Etan Kissling efbd4e395a
avoid sending redundant LC finality updates (#4546)
When the epoch boundary block is missed, we incorrectly assume that the
next couple blocks improve finality, leading to repeated pushes of the
same light client finality update and incorrectly ignoring some gossip.
2023-01-24 17:44:55 +00:00
tersec aacc8d702d
remove Nim 1.2-compatible `push raise`s and update copyright notice years (#4528) 2023-01-20 14:14:37 +00:00
tersec 819e007689
exit/validatorchange pool includes BLS to execution messages; REST support for new pool (#4519)
* exit/validatorchange pool includes BLS to execution messages; REST
support for new pool

* catch failed individual futures

* increase BLS changes bound and keep BLS seen consistent with subpool

* deque capacities should be powers of 2
2023-01-19 22:00:40 +00:00
tersec 073c544f0c
automated update from v1.3.0-rc.0 to v1.3.0-rc.1 consensus spec URLs (#4517) 2023-01-17 16:10:52 +00:00
tersec 9b0868033e
use GitHub Actions cache and upload-artifact v3 (#4516) 2023-01-17 15:45:04 +00:00
Etan Kissling 609227559f
LC data fork cleanup (#4506)
Distinguish between those code locations that need to be updated on each
light client data format change, and those others that should generally
be fine, as long as a valid light client object is processed.

The former are tagged with static assert for `LightClientDataFork.high`.

The latter are changed to `lcDataFork > LightClientDataFork.None` to
indicate that they depend only on presence of any valid object.
Also bundled a few minor cleanups and fixes.

Also add `Forky` type for `LightClientStore` and minor fixes / cleanups.
2023-01-14 22:19:50 +01:00
Etan Kissling 2324136552
add `LightClientHeader` wrapper (#4481)
The light client data structures were changed to accommodate additional
fields in future forks (e.g., to also hold execution data).

There is a minor change to the JSON serialization, where the `header`
properties are now nested inside a `LightClientHeader`.
The SSZ serialization remains compatible.

See https://github.com/ethereum/consensus-specs/pull/3190
and https://github.com/ethereum/beacon-APIs/pull/287
2023-01-13 16:46:35 +01:00
Etan Kissling 7e276937dc
make LC data fork aware (#4493)
In a future fork, light client data will be extended with execution info
to support more use cases. To anticipate such an upgrade, introduce
`Forky` and `Forked` types, and ready the database schema.
Because the mapping of sync committee periods to fork versions is not
necessarily unique (fork schedule not in sync with period boundaries),
an additional column is added to `period` -> `LightClientUpdate` table.
2023-01-12 18:11:38 +01:00
Jacek Sieka 6bfc766629
drop subset sync contributions in gossip (#4490)
* correctly report ignored contributions in metrics
* avoid counting subset contributions in vmon (bring in line with
attestation aggregates)
* avoid signature checks for subset attestations

A being a non-strict subset is a sufficient condition to ignore.
2023-01-12 15:08:08 +01:00
tersec 4f9064a153
consensus spec ref URL updates (#4496) 2023-01-12 13:33:14 +00:00
Jacek Sieka ba3db7aa5a
spec: Option -> Opt (#4488) 2023-01-11 12:29:21 +00:00
tersec e28e1aeec8
a few consensus spec ref URL updates (#4483) 2023-01-10 16:14:17 +00:00
Etan Kissling 0590be7afe
add `num_active_participants` helpers for `SyncAggregate` (#4478)
Introduce `num_active_participants` helper function to reduce visibility
of low-level `countOnes` function and reduce code duplication.
2023-01-10 11:26:25 +00:00
tersec 2dd3cd786f
consensus spec ref URL update v1.3.0-{alpha.2,rc.0}; copyright year update (#4477) 2023-01-09 22:44:44 +00:00
henridf 64878888bd
Blob storage (#4454)
* Blob storage

* fix indentation

* Fix build (none->Opt.none)

* putBlobs -> putBlobsSidecar

* getBlobs -> getBlobsSidecar

* Check blob correctness when storing a backfill block

* Blobs table: rename and conditionally create

* Check block<->blob match in storeBackfillBlock

* Use when .. toFork() to condition on type

* Check blob viability in block_processor.storeBlock()

* Fix build

* Review feedback
2023-01-09 18:42:10 +00:00
Jacek Sieka 0ba9fc4ede
History pruning (fixes #4419) (#4445)
Introduce (optional) pruning of historical data - a pruned node will
continue to answer queries for historical data up to
`MIN_EPOCHS_FOR_BLOCK_REQUESTS` epochs, or roughly 5 months, capping
typical database usage at around 60-70gb.

To enable pruning, add `--history=prune` to the command line - on the
first start, old data will be cleared (which may take a while) - after
that, data is pruned continuously.

When pruning an existing database, the database will not shrink -
instead, the freed space is recycled as the node continues to run - to
free up space, perform a trusted node sync with a fresh database.

When switching on archive mode in a pruned node, history is retained
from that point onwards.

History pruning is scheduled to be enabled by default in a future
release.

In this PR, `minimal` mode from #4419 is not implemented meaning
retention periods for states and blocks are always the same - depending
on user demand, a future PR may implement `minimal` as well.
2023-01-07 10:02:15 +00:00
tersec 47cb0f7991
capella forkchoiceUpdated support (#4462)
* capella forkchoiceUpdated support

* match V2 fcU with V2 getPayload
2023-01-06 22:01:10 +01:00
Jacek Sieka 7c2ed5c609
Always-on optimistic mode (#4458)
With https://github.com/status-im/nimbus-eth2/pull/4420 implemented, the
checks that we perform are equivalent to those of a `SYNCING` EL - as
such, we can treat missing EL the same as SYNCING and proceed with an
optimistic sync.

This mode of operation significantly speeds up recovery after an offline
EL event because the CL is already synced and can immediately inform the
EL of the latest head.

It also allows using a beacon node for consensus archival queries
without an execution client.

* deprecate `--optimistic` flag
* log block details on EL error, soften log level because we can now
continue to operate
* `UnviableFork` -> `Invalid` when block hash verification fails -
failed hash verification is not a fork-related block issue
2023-01-04 15:51:14 +00:00
Jacek Sieka aff0505807
Allow reindexing to start from an arbitrary state (#4437)
When not backfilling all the way to genesis (#4421), it becomes more
useful to start rebuilding the historical indices from an arbitrary
starting point.

To rebuild the index from non-genesis, a state and an unbroken block
history is needed - here, we allow loading the state from an era file
and recreating the history from there onwards.

* speed up partial era state loading
2023-01-03 20:37:09 +01:00
tersec 121c66ad3e
fix getForkedBlock to read EIP4844 and Capella blocks (#4450)
* fix getForkedBlock to read EIP4844 and Capella blocks

* update geth develop URLs
2022-12-30 21:59:21 +00:00
Jacek Sieka 75c7195bfd
Backfill only up to MIN_EPOCHS_FOR_BLOCK_REQUESTS blocks (#4421)
When backfilling, we only need to download blocks that are newer than
MIN_EPOCHS_FOR_BLOCK_REQUESTS - the rest cannot reliably be fetched from
the network and does not have to be provided to others.

This change affects only trusted-node-synced clients - genesis sync
continues to work as before (because it needs to construct a state by
building it from genesis).

Those wishing to complete a backfill should do so with era files
instead.
2022-12-23 08:42:55 +01:00
Etan Kissling 2ac7609259
display owl on Capella transition (#4442)
Trigger ANSI art on upgrade to Capella, similar to the merge.
Future extension could log blinking art when user successfully managed
to get BLS to Execution change included into a block for a validator.

Art created by http://beatscribe.com/ (beatscribe#1008 on Discord)
2022-12-21 13:30:24 +01:00
tersec bb4ea37baa
update EF consensus spec URLs from v1.3.0-alpha.1 to v1.3.0-alpha.2 (#4432) 2022-12-15 12:15:12 +00:00
tersec e3c062189a
eip4844 `block_sim` (#4405) 2022-12-09 21:39:11 +00:00
tersec dee5af58d6
eip4844 light client tests; avoid case object out-of-bound array reads (#4404) 2022-12-08 17:21:53 +01:00
tersec 2932d3b808
extent `BeaconStateFork` enum (#4396) 2022-12-07 16:47:23 +00:00
zah d30cb8baf1
Support for obtaining deposit snapshots during trustedNodeSync (#4303)
Other changes:

* More optimal search for TTD block.

* Add timeouts to all REST requests during trusted node sync.
  Fixes #4037

* Removed support for storing a deposit snapshot in the network
  metadata.
2022-12-07 12:24:51 +02:00
tersec 031780b612
eip4844 operations tests (#4392) 2022-12-06 12:40:13 +00:00
tersec 415b11aa67
EIP4844 tweaks to pass SSZ consensus object tests (#4390) 2022-12-05 21:36:53 +00:00
henridf f0329b2212
Types and scaffolding for EIP-4844 (#4365)
* Types and scaffolding for EIP-4844

This commit adds the EIP-4844 spec types, and fills in
scaffolding/boilerplate for the use of these types across the repo.

None of the actual EIP-4844 logic is introduced yet.

This follows the pattern used by @tersec when introducing Capella (#4276).

* use eth2-networks fork

* review feedback: add static check EIP4844_FORK_EPOCH == FAR_FUTURE_EPOCH

* review feedback: remove EIP4844 from /eth/v1/config/spec response

* Cleanup / review feedback

* Fix REST test
2022-12-05 16:29:09 +00:00
tersec 5c16062de9
remove all but truly stub support for `SHARDING_FORK_{EPOCH,VERSION}` (#4385) 2022-12-02 13:33:18 +01:00
zah 7c783644a2
Fix and document some usages of defaultRuntimeConfig (#4147)
Other changes:

* Make the light client store compatible with phase0-only networks
  and simulations
2022-12-01 11:25:21 +00:00
tersec 474b0d8502
`withUpdatedState` injects `updatedState` rather than `state` template (#4375) 2022-11-30 16:37:23 +02:00
tersec df54470b13
verify `CAPELLA_FORK_EPOCH == FAR_FUTURE_EPOCH` for supported networks (#4369) 2022-11-29 09:12:25 +01:00
tersec 898684c175
REST server shouldn't create validation objects when optimistically synced (#4361) 2022-11-25 22:49:31 +00:00
tersec 61c5ac32d8
automated consensus spec ref URL update to v1.3.0-alpha.1 (#4354) 2022-11-24 19:07:02 +00:00
tersec ec443601eb
implement capellaImplementationMissing points; don't track not-active validator duties (#4340)
* implement several capellaImplementationMissing points

* don't register validator activity for not-active validators

* don't check validator indices already coming out of committees which exist; must be active validators, or else other deeper bugs
2022-11-22 13:56:05 +02:00
Eugene Kabanov eb661565ed
Per-validator doppelganger protection. (#4304)
* Initial commit.

* NextAttestationEntry type.

* Add doppelgangerCheck and actual check.

* Recover deleted check.

* Remove NextAttestainEntry changes.

* More cleanups for NextAttestationEntry.

* Address review comments.

* Remove GENESIS_EPOCH specific check branch.

* Decrease number of full epochs for doppelganger check in VC.

Co-authored-by: zah <zahary@status.im>
2022-11-20 15:55:43 +02:00
tersec b3f6be71d5
refactor `makeBeaconBlock`; some capella support for `ncli_db` and `wss_sim` (#4321) 2022-11-11 15:37:43 +01:00
tersec 35b1104bea
`block_sim` runs capella by default (#4315) 2022-11-11 10:17:27 +00:00
Etan Kissling 48994f67d3
rename `BlockError` -> `VerifierError` (#4310)
We currently use `BlockError` for both beacon blocks and LC objects.
In light of EIP4844, we will likely also use it for blob sidecars.
To avoid confusion, renaming it to a more generic `VerifierError`,
and update its documentation to be more generic.

To avoid long lines as a followup, also renaming the `block_processor`'s
`BlockProcessingCompleted.completed`->`ProcessingStatus.completed` and
`BlockProcessingCompleted.notCompleted`->`ProcessingStatus.notCompleted`
2022-11-10 17:40:27 +00:00
Jacek Sieka 09ade6d33d
Make trusted node sync era-aware (#4283)
This PR removes a bunch of code to make TNS aware of era files, avoiding
a duplicated backfill when era files are available.

* reuse chaindag for loading backfill state, replacing the TNS homebrew
* fix era block iteration to skip empty slots
* add tests for `can_advance_slots`
2022-11-10 10:44:47 +00:00
tersec 90eb2ccb20
database and fork choice test runner support for capella (#4309) 2022-11-09 17:32:10 +00:00
Jacek Sieka 8297b962cc
Fix VC-based sync subnet subscriptions (#4293)
* move duty tracking code to `ActionTracker`
* fix earlier duties overwriting later ones
* re-run subnet selection when new duty appears
* log upcoming duties as soon as they're known (vs 4 epochs before)
2022-11-08 12:43:38 +01:00
tersec 909c095e64
initial automated v1.2.0 -> v1.3.0-alpha.0 consensus spec URL update (#4296) 2022-11-08 02:37:28 +00:00
tersec 5b46f0b723
add Capella support to Forked* (#4276)
* add Capella support to Forked*

* remove cruft

* add `OnForkyBlockAdded`
2022-11-02 16:23:30 +00:00