72 Commits

Author SHA1 Message Date
Etan Kissling
7c53841cd8
Revert "Revert "fix checkpoint block potentially not getting backfilled into DB (#5863)" (#5871)" (#5875)
This reverts commit 1575478b721b6f62d6466a226b91188bdf97e00b.
2024-02-09 20:44:54 +01:00
tersec
1575478b72
Revert "fix checkpoint block potentially not getting backfilled into DB (#5863)" (#5871)
This reverts commit 65e6f892deb5d9ff4399a0840a90788726024008.
2024-02-09 12:49:07 +00:00
Etan Kissling
65e6f892de
fix checkpoint block potentially not getting backfilled into DB (#5863)
When using checkpoint sync, only checkpoint state is available, block is
not downloaded and backfilled later.

`dag.backfill` tracks latest filled `slot`, and latest `parent_root` for
which no block has been synced yet.

In checkpoint sync, this assumption is broken, because there, the start
`dag.backfill.slot` is set based on checkpoint state slot, and the block
is also not available.

However, sync manager in backward mode also requests `dag.backfill.slot`
and `block_clearance` then backfills the checkpoint block once it is
synced. But, there is no guarantee that a peer ever sends us that block.
They could send us all parent blocks and solely omit the checkpoint
block itself. In that situation, we would accept the parent blocks and
advance `dag.backfill`, and subsequently never request the checkpoint
block again, resulting in gap inside blocks DB that is never filled.

To mitigate that, the assumption is restored that `dag.backfill.slot`
is the latest filled `slot`, and `dag.backfill.parent_root` is the next
block that needs to be synced. By setting `slot` to `tail.slot + 1` and
`parent_root` to `tail.root`, we put a fake summary into `dag.backfill`
so that `block_clearance` only proceeds once checkpoint block exists.
2024-02-09 11:20:36 +01:00
Etan Kissling
94ba0a9bd1
consider block availability when initializing LC data collector (#5860)
When using checkpoint sync, the initial block is missing in the DB.
Update the LC data collector initialization to account for that,
avoiding a spurious error message when it is incorrectly accessed:

```
ERR 2024-02-07 11:21:55.416+01:00 Block failed to load unexpectedly          topics="chaindag_lc" bid=d30517a7:8257504 tail=8257504
```

Also fixes a regression from #5691 that resulted in similar messages
while importing the first few blocks after checkpoint sync.

Thanks to @arnetheduck for reporting this.
2024-02-07 18:03:19 +00:00
Etan Kissling
be73ce2e9a
import finalized head LC bootstrap on launch (#5775)
If the initial state replays cover the finalized head, import matching
`LightClientBootstrap` into database.

This also addresses this error when light client requests bootstrap from
the genesis slot on networks that launch with Altair enabled.

```
{"lvl":"DBG","ts":"2023-10-04 11:17:49.665+00:00","msg":"LC bootstrap unavailable: Sync committee branch not cached","topics":"chaindag_lc","slot":0}
```
2024-01-18 22:51:26 +00:00
Etan Kissling
7db95f047b
track latest LightClientUpdate only once fork choice selects it (#5691)
Instead of tracking the latest `LightClientUpdate` across all branches,
track the latest one on the current branch as selected by fork choice.
2024-01-03 23:36:05 +01:00
tersec
115ffa70eb
rm unused code (#5623) 2023-11-25 12:09:18 +00:00
Etan Kissling
b0b8f21223
fix LC data import --only-new regression (#5616)
Corrects LC server startup logic for `LightClientDataImportMode.OnlyNew`
after regression in #5607.
2023-11-22 14:05:31 +01:00
Etan Kissling
c33dd2c170
restrict best LC update collection to canonical blocks (#5613)
Simplify best `LightClientUpdate` collection by tracking only canonical
data instead of tracking the best update across all branches within the
sync committee period.

- https://github.com/ethereum/consensus-specs/pull/3553
2023-11-21 23:51:05 +01:00
Etan Kissling
d8144c6de1
cache new LC data after creating new LC updates (#5607)
When creating new LC updates, information about the parent block's post
state must be available (cached), but information about current block's
post state is not yet required. Caching information about the current
block's post state can be delayed, simplifying the LC data collection
logic a bit and allowing more future flexibility with the cache design.
2023-11-18 01:17:26 +00:00
Etan Kissling
b179cb71a0
clean up XDeclaredButNotUsed for light client (#5606)
Handle remaining `XDeclaredButNotUsed` warnings for light client.
2023-11-18 00:03:27 +01:00
Etan Kissling
b9a693d5fe
send LC finality update on event stream on supermajority (#5602)
When new finality is reached without supermajority sync committee
support, trigger another event push on beacon-API and libp2p once
the finality gains supermajority support.

- https://github.com/ethereum/consensus-specs/pull/3549
2023-11-16 19:57:15 -08:00
Etan Kissling
0919ff05c9
reduce code duplication when collecting LC data (#5601)
Replace sections that need to be maintained with every `ConsensusFork`
related to LC data collection with a generic logic that keeps working
when unrelated parts of Ethereum change.
2023-11-15 03:21:03 +01:00
Etan Kissling
d9031d97dc
rename _INDEX generalized index constants to _GINDEX (#5583)
Rename constants for consistency with upcoming spec adjustment:

- https://github.com/ethereum/consensus-specs/pull/3537
2023-11-09 12:54:44 +00:00
tersec
556d5e7114
rm unused code (#5538) 2023-11-01 05:53:09 +01:00
Etan Kissling
297c768816
reduce redundant zero initialization for LC data objects (#5479)
Directly initialize `ForkedLightClientObj` instead of separately first
 setting the `kind` (initializing everything to zero) and then assigning
the forky data after that.
2023-10-04 18:11:45 +02:00
Etan Kissling
e7bc41e005
blck --> forkyBlck when using withBlck / withStateAndBlck (#5451)
For symmetry with `forkyState` when using `withState`, and to avoid
problems with shadowing of `blck` when using `withBlck` in `template`,
also rename the injected `blck` to `forkyBlck`.

- https://github.com/nim-lang/Nim/issues/22698
2023-09-21 12:49:14 +02:00
Etan Kissling
fd8e86972d
rename lcDataForkAtStateFork > lcDataForkAtConsensusFork (#4726) 2023-03-11 20:09:21 +00:00
Etan Kissling
ad118cd354
rename stateFork > consensusFork (#4718)
Just the variable, not yet `lcDataForkAtStateFork` / `atStateFork`.

- Shorten comment in `light_client.nim` to keep line width
- Do not rename `stateFork` mention in `runProposalForkchoiceUpdated`.
- Do not rename `stateFork` in `getStateField(dag.headState, fork)`

Rest is just a mechanical mass replace
2023-03-11 00:35:52 +00:00
Etan Kissling
e499c709f4
avoid pruning LC data pre dag.tail.slot (#4702)
When using `--history=prune`, `dag.tail.slot` may advance beyond the
configured light client data retention period. Update the LC logic so
that the `dag.tail.slot` is no longer considered for LC pruning.
It is still considered to check whether new data can be produced.
2023-03-08 19:59:00 +01:00
tersec
e3d96ef147
rename most eip4844Data to denebData (#4693) 2023-03-04 22:23:52 +00:00
tersec
ea060de6d4
more eip4844 -> deneb module references (#4690) 2023-03-02 21:09:24 +01:00
tersec
2dd2a782d9
disable BareExcept warnings (#4662)
* disable BareExcept warnings

* BareExcept only exists in >= 1.6.11
2023-02-24 19:50:46 +00:00
tersec
0fb726c420
BeaconStateFork/BeaconBlockFork -> ConsensusFork (#4560)
* `BeaconStateFork/BeaconBlockFork` -> `ConsensusFork`

* revert unrelated change

* revert unrelated changes

* update test summaries
2023-01-28 19:53:41 +00:00
Etan Kissling
9f279e2933
cache LC headers and sync committees for bootstrap (#4499)
To allow LC data retention longer than the one for historic states,
introduce persistent DB caches for `current_sync_committee` and
`LightClientHeader` for finalized epoch boundary blocks.
This way, historic `LightClientBootstrap` requests may still be honored
even after pruning. Note that historic `LightClientUpdate` requests are
already answered using fully persisted objects, so don't need changes.

Sync committees and headers are cached on finalization of new data.
For existing data, info is lazily cached on first access.

Co-authored-by: Jacek Sieka <jacek@status.im>
2023-01-27 10:44:57 +01:00
Etan Kissling
efbd4e395a
avoid sending redundant LC finality updates (#4546)
When the epoch boundary block is missed, we incorrectly assume that the
next couple blocks improve finality, leading to repeated pushes of the
same light client finality update and incorrectly ignoring some gossip.
2023-01-24 17:44:55 +00:00
tersec
aacc8d702d
remove Nim 1.2-compatible push raises and update copyright notice years (#4528) 2023-01-20 14:14:37 +00:00
Etan Kissling
609227559f
LC data fork cleanup (#4506)
Distinguish between those code locations that need to be updated on each
light client data format change, and those others that should generally
be fine, as long as a valid light client object is processed.

The former are tagged with static assert for `LightClientDataFork.high`.

The latter are changed to `lcDataFork > LightClientDataFork.None` to
indicate that they depend only on presence of any valid object.
Also bundled a few minor cleanups and fixes.

Also add `Forky` type for `LightClientStore` and minor fixes / cleanups.
2023-01-14 22:19:50 +01:00
Etan Kissling
2324136552
add LightClientHeader wrapper (#4481)
The light client data structures were changed to accommodate additional
fields in future forks (e.g., to also hold execution data).

There is a minor change to the JSON serialization, where the `header`
properties are now nested inside a `LightClientHeader`.
The SSZ serialization remains compatible.

See https://github.com/ethereum/consensus-specs/pull/3190
and https://github.com/ethereum/beacon-APIs/pull/287
2023-01-13 16:46:35 +01:00
Etan Kissling
7e276937dc
make LC data fork aware (#4493)
In a future fork, light client data will be extended with execution info
to support more use cases. To anticipate such an upgrade, introduce
`Forky` and `Forked` types, and ready the database schema.
Because the mapping of sync committee periods to fork versions is not
necessarily unique (fork schedule not in sync with period boundaries),
an additional column is added to `period` -> `LightClientUpdate` table.
2023-01-12 18:11:38 +01:00
Etan Kissling
0590be7afe
add num_active_participants helpers for SyncAggregate (#4478)
Introduce `num_active_participants` helper function to reduce visibility
of low-level `countOnes` function and reduce code duplication.
2023-01-10 11:26:25 +00:00
tersec
dee5af58d6
eip4844 light client tests; avoid case object out-of-bound array reads (#4404) 2022-12-08 17:21:53 +01:00
tersec
2932d3b808
extent BeaconStateFork enum (#4396) 2022-12-07 16:47:23 +00:00
tersec
031780b612
eip4844 operations tests (#4392) 2022-12-06 12:40:13 +00:00
zah
7c783644a2
Fix and document some usages of defaultRuntimeConfig (#4147)
Other changes:

* Make the light client store compatible with phase0-only networks
  and simulations
2022-12-01 11:25:21 +00:00
tersec
474b0d8502
withUpdatedState injects updatedState rather than state template (#4375) 2022-11-30 16:37:23 +02:00
tersec
ec443601eb
implement capellaImplementationMissing points; don't track not-active validator duties (#4340)
* implement several capellaImplementationMissing points

* don't register validator activity for not-active validators

* don't check validator indices already coming out of committees which exist; must be active validators, or else other deeper bugs
2022-11-22 13:56:05 +02:00
tersec
b3f6be71d5
refactor makeBeaconBlock; some capella support for ncli_db and wss_sim (#4321) 2022-11-11 15:37:43 +01:00
tersec
5b46f0b723
add Capella support to Forked* (#4276)
* add Capella support to Forked*

* remove cruft

* add `OnForkyBlockAdded`
2022-11-02 16:23:30 +00:00
tersec
0a4aa5fdb3
switch withStateAndBlck usage to forkyState; rm unused tests/mocking/ modules (#4206) 2022-10-03 13:08:50 +00:00
Etan Kissling
c9f69fc38e
fix best LC updates when backfilling (#4195)
When backfilling LC updates (`--light-client-data-import-mode=full`),
the highest participation update is computed without ensuring that the
finalized header is in the same period. Updates sharing same period for
both finalized and attested headers should be preferred.
Fixes a bug leading to suboptimal update selection.
2022-09-29 07:55:04 +02:00
tersec
b60456fdf3
withState: state -> forkyState (#4038) 2022-08-26 22:47:40 +00:00
Etan Kissling
74dc388ad9
do not prune LC data by default (#4008)
Aligns the default retention policy for LC data with the one for blocks.
Minimum spec requirement for both blocks and LC data is ~5 months.
Additional use cases are better supported by retaining data for longer.
2022-08-21 11:24:59 +02:00
Etan Kissling
5c8e58ea23
update LC spec references for v1.2.0-rc.2 (#3982)
Updates light client spec references for latest spec (no more `vFuture`)
2022-08-17 19:47:06 +00:00
Miran
dfd4afc9f2
compatibility with Nim 1.4+ (#3888) 2022-07-29 10:53:42 +00:00
tersec
ce6cbd84e2
rename verifyFinalization internal flag to strictVerification (#3866)
* rename verifyFinalization internal flag to strictVerification

* Update beacon_chain/extras.nim

Co-authored-by: Etan Kissling <etan@status.im>

Co-authored-by: Etan Kissling <etan@status.im>
2022-07-13 13:48:09 +00:00
Etan Kissling
aff53e962f
merge LC db into main BN db (#3832)
* merge LC db into main BN db

To treat derived LC data similar to derived state caches, merge it into
the main beacon node DB.

* shorten table names, group with lc prefix
2022-07-04 23:46:32 +03:00
Etan Kissling
499abd927f
persist LC data across restarts (#3823)
* persist LC data across restarts

With the Altair spec `LightClientUpdate` structure taking its final form
it is finally possible to persist LC data across restarts without having
to worry about data migration due to spec changes. A separate `lcdataV1`
database is created in the `caches` subdirectory to hold known LC data.
A full database with default settings (129 periods) uses <15 MB disk.

* extend LC data DB rationale

* wording

* add `isSupportedBySQLite` helper and explicit return

* remove redundant `return`
2022-06-30 13:04:39 +00:00
Jacek Sieka
c145916414
cleanups (#3819)
* avoid circular panda imports
* move deposit merkleization helpers to spec/
* normalize validator signature helpers to spec names / params
* remove redundant functions for remote signing
2022-06-29 18:53:59 +02:00
Etan Kissling
bc1cc8f643
encapsulate LC config into one type (#3817)
Separate LC initialization options from the main ChainDAGRef options to
allow ChainDAGRef to treat them as opaque and reduce risk for conflicts
when extending those options in the future.
2022-06-28 22:52:29 +02:00