Commit Graph

319 Commits

Author SHA1 Message Date
tersec 6b8061b5d6
automated consensus spec URL updating to v1.5.0-alpha.2 () 2024-05-09 05:03:10 +00:00
Etan Kissling 9bffc51cf6
allow frontfilling finalized CP block from era file ()
Add support for using era file for the initial checkpoint block.
This should also avoid an error when the beacon node is restarted
before the backfill process has made any progress ().
2024-04-10 14:11:07 +02:00
tersec 27921406e9
remove some debugRaiseAsserts and fill in Electra functionality () 2024-04-06 15:11:47 +02:00
Etan Kissling df0ff5f0fb
fix initialization of sync committee cache after loading non-epoch state ()
When initializing from a state that's not aligned to an epoch boundary,
an earlier state is loaded that's epoch aligned, and subsequently topped
up with the missing blocks. `dag.headSyncCommittee` is initialized prior
to topping up the missing blocks, though. If the sync committee changes
while applying the blocks (e.g., a sync committee period boundary hits),
the cached information becomes unlinked from `dag.head`, leading to
valid blocks based on that chain being rejected. To fix this, move cache
initialization after the top up with blocks. This has been observed on
Goerli by initializing from 7919502 and attempting to top up 7920111.
The block gets rejected with an invalid state root on nodes that have
restarted after setting 7920111 as head, while it gets accepted by all
other nodes. Error message is `block: state root verification failed`.
The incorrect initialization behaviour was introduced in , before
which the sync committee cache was initialized after applying blocks.
2024-04-03 23:03:06 +02:00
tersec 7fa32b7f02
add Electra to ConsensusFork enum ()
* add Electra to ConsensusFork enum

* fix gnosis check
2024-04-03 16:43:43 +02:00
Etan Kissling 6a318d0f1a
avoid rejecting empty era file in verification ()
`batchVerify`'s precondition is a non-empty signature list:

```nim
  if input.len == 0:
    # Spec precondition
    return false
```

This means that in eras without any blocks (as has happened on Goerli),
calling it leads to era files being reported as invalid.
2024-04-03 10:06:21 +02:00
Etan Kissling 2dbe24c740
move split view catchup to research branch ()
Using a dedicated branch for researching the effectiveness of split view
scenario handling simplifies testing and avoids having partial work on
`unstable`. If we want, we can reintroduce it under a `--debug` flag at
a later time. But for now, Goerli is a rare opoprtunity to test this,
maybe just for another week or so.

- https://github.com/status-im/infra-nimbus/pull/179
2024-03-25 19:09:31 +01:00
Etan Kissling fc9bc1da3a
add branch discovery module for supporting chain stall situation ()
In split view situation, the canonical chain may only be served by a
tiny amount of peers, and branches may span long durations. Minority
branches may still have a large weight from attestations and should
be discovered. To assist with that, add a branch discovery module that
assists in such a situation by specifically targeting peers with unknown
histories and downloading from them, in addition to sync manager work
which handles popular branches.
2024-03-24 08:41:47 +00:00
Etan Kissling 66a9304fea
use separate state when catching up to perform validator duties ()
There are situations where all states in the `blockchain_dag` are
occupied and cannot be borrowed.

- headState: Many assumptions in the code that it cannot be advanced
- clearanceState: Resets every time a new block gets imported, including
  blocks from non-canonical branches
- epochRefState: Used even more frequently than clearanceState

This means that during the catch-up mechanic where the head state is
slowly advanced to wall clock to catch up on validator duties in the
situation where the canonical head is way behind non-canonical heads,
we cannot use any of the three existing states. In that situation,
Nimbus already consumes an increased amount of memory due to all the
`BlockRef`, fork choice states and so on, so experience is degraded.
It seems reasonable to allocate a fourth state temporarily during that
mechanic, until a new proposal could be made on the canonical chain.

Note that currently, on `unstable`, proposals _do_ happen every couple
hours because sync manager doesn't manage to discover additional heads
in a split-view scenario on Goerli. However, with the branch discovery
module, new blocks are discovered all the time, and the clearanceState
may no longer be borrowed as it is reset to different branch too often.

The extra state could also find other uses in the future, e.g., for
incremental computations as in reindexing the database, or online
collection of historical light client data.
2024-03-24 07:18:33 +01:00
Etan Kissling 33e34ee8bd
handle case of unreachable block in `is_optimstic` helper ()
* handle case of unreachable block in `is_optimstic` helper

When a non-canonical block is still in the DB, it can be accessed via
`BlockId`, but `BlockRef` may be unavailable if the block was not
properly cleaned when it got orphaned. Report it as optimistic.

* `template` -> `func`
2024-03-22 22:50:21 +00:00
Etan Kissling 035ca015e6
continue validator duties if chain does not progress for a long time ()
Nimbus currently stops performing validator duties if the blockchain
does not progress for `node.config.syncHorizon` slots. This means that
the chain won't recover because no new blocks are proposed. To fix that,
continue performing validator duties if no progress is registered for a
long time, and none of our peers is indicating any progress.
2024-03-20 03:23:53 +01:00
Etan Kissling 595d110b37
avoid blocking deep reorgs > 64 epochs ()
On Goerli there are some instances of long streaks of empty epochs due
to different branches being built in parallel. They sometimes lead to
`Request for pruned historical state` logs requiring a BN restart to
resolve. Avoid that by trying to restore states from the entire non-
finalized history, to avoid losing sync in such situtions.
2024-03-19 14:21:25 +01:00
tersec 0a6d189161
automated consensus spec URL updating to v1.4.0 () 2024-03-14 07:26:36 +01:00
tersec 2a13c09615
add proposer reward accounting to block transitions ()
* add proposer reward accounting to block transitions

* Update beacon_chain/spec/state_transition_block.nim

Co-authored-by: Etan Kissling <etan@status.im>

---------

Co-authored-by: Etan Kissling <etan@status.im>
2024-03-04 17:00:46 +00:00
tersec a4f4a35845
Revert "initial Electra support skeleton" ()
* Revert "initial Electra support skeleton ()"

This reverts commit d09bf3b587.

* Update test_signing_node.nim
2024-02-25 19:42:44 +00:00
tersec d09bf3b587
initial Electra support skeleton () 2024-02-24 13:44:15 +00:00
tersec c73d7c6f6f
automated consensus spec URL updating to v1.4.0-beta.7 () 2024-02-21 19:44:48 +00:00
Etan Kissling 88045a91cd
rename new timing metrics, as `_total` suffix is implicit ()
* track latest duration instead of total in new timing metrics

Change `db_checkpoint_seconds` and `state_replay_seconds` metrics to
record the latest duration instead of the total. `nim-metrics` already
synthesizes a `_total` metric from these implicitly.

* still have to use inc, metrics only synthesizes the name not the sum

* prefix with `beacon_dag`
2024-02-20 20:34:41 +01:00
Jacek Sieka 8d465a7d8c
vmon: Missed block metric ()
Validator monitoring gained 2 new metrics for tracking when blocks are
included or not on the head chain.

Similar to attestations, if the block is produced in epoch N, reporting
will use the state when switching to epoch N+2 to do the reporting (so
as to reasonably stabilise the block inclusion in the face of reorgs).
2024-02-20 06:40:18 +02:00
Etan Kissling 92197ce690
add metric for database checkpoint duration ()
Database checkpointing can take seconds, e.g., while Geth is syncing.
Add a debug log + metric for it, and also info log if it takes longer
than 250ms, same as for the existing `State replayed` log. If the log
shows up for a user while the system is not overloaded, it may point
to slow disk speed or thermal issue.
2024-02-19 11:00:11 +01:00
Jacek Sieka afdfe302f3
state loading optimizations ()
* compute post-merge randao mix without loading state
* avoid copying state on shuffling computation and compute epochref
* speed up state copy for block production
2024-02-12 15:58:55 +01:00
Etan Kissling 9593ef74b8
do not cache zero block hash if block unavailable ()
With checkpoint sync, the checkpoint block is typically unavailable at
the start, and only backfilled later. To avoid treating it as having
zero hash, execution disabled in some contexts, wrap the result of
`loadExecutionBlockHash` in `Opt` and handle block hash being unknown.

---------

Co-authored-by: Jacek Sieka <jacek@status.im>
2024-02-09 22:10:38 +00:00
Etan Kissling 7c53841cd8
Revert "Revert "fix checkpoint block potentially not getting backfilled into DB ()" ()" ()
This reverts commit 1575478b72.
2024-02-09 20:44:54 +01:00
Etan Kissling f2d92729a2
reduce verbosity of `Got request for pre-backfill slot` ()
When syncing, we log a notice each time someone asks us for a block that
we haven't backfilled yet. This is quite verbose and not unexpected,
because the status message does not allow indicating backfill progress.
2024-02-09 20:32:31 +01:00
tersec 1575478b72
Revert "fix checkpoint block potentially not getting backfilled into DB ()" ()
This reverts commit 65e6f892de.
2024-02-09 12:49:07 +00:00
Etan Kissling 65e6f892de
fix checkpoint block potentially not getting backfilled into DB ()
When using checkpoint sync, only checkpoint state is available, block is
not downloaded and backfilled later.

`dag.backfill` tracks latest filled `slot`, and latest `parent_root` for
which no block has been synced yet.

In checkpoint sync, this assumption is broken, because there, the start
`dag.backfill.slot` is set based on checkpoint state slot, and the block
is also not available.

However, sync manager in backward mode also requests `dag.backfill.slot`
and `block_clearance` then backfills the checkpoint block once it is
synced. But, there is no guarantee that a peer ever sends us that block.
They could send us all parent blocks and solely omit the checkpoint
block itself. In that situation, we would accept the parent blocks and
advance `dag.backfill`, and subsequently never request the checkpoint
block again, resulting in gap inside blocks DB that is never filled.

To mitigate that, the assumption is restored that `dag.backfill.slot`
is the latest filled `slot`, and `dag.backfill.parent_root` is the next
block that needs to be synced. By setting `slot` to `tail.slot + 1` and
`parent_root` to `tail.root`, we put a fake summary into `dag.backfill`
so that `block_clearance` only proceeds once checkpoint block exists.
2024-02-09 11:20:36 +01:00
Etan Kissling 4266e16835
allow `getBlockIdAtSlot` to answer queries from available states ()
After checkpoint sync, historical block IDs cannot yet be queried.
However, they are needed to compute dependent roots of `ShufflingRef`.
To allow lookup, enable `getBlockIdAtSlot` to answer from compatible
states in memory; as long as they descend from the finalized checkpoint
and the requested slot is sufficiently recent, `block_roots` contains
everything to recover `BlockSlotId` up to `SLOTS_PER_HISTORICAL_ROOT`.
This is similar to how `attester_dependent_root` etc. are computed.

This accelerates the first couple minutes of checkpoint sync on Mainnet,
especially the time until finality advances past the synced checkpoint.
2024-02-09 11:13:00 +01:00
tersec 6c53dc1e11
automated consensus spec URL updating to v1.4.0-beta.6 () 2024-01-20 11:19:47 +00:00
tersec cf1bec7670
update some deprecated stew/results to results imports () 2024-01-16 22:37:14 +00:00
Jacek Sieka 62cbdeefc5
verify `genesis_time` more strictly (fixes ) ()
Bogus values lead to crashes down the line when timers overflow
2024-01-06 15:26:56 +01:00
Jacek Sieka 4a56faa579
era: fix verifier at empty slots ()
* era: fix verifier at empty slots

* avoid returning zero-byte block data to REST/p2p when loading era
files

* fix local test
2023-12-05 07:55:25 +01:00
tersec 9efb2958ec
automated consensus spec URL updating to v1.4.0-beta.5 () 2023-12-05 03:34:45 +01:00
Etan Kissling 8cea8af620
fix startup after BN exited between head and finalized blocks updates ()
When the BN exits after writing new `head` to database, but before
completing the `updateFinalizedBlocks` call, the database is slightly
inconsistent due to the partial write. We currently fail to start up
after that. Fix that by catching up on partial `updateFinalizedBlocks`
tasks on start up, and add a test for this edge case.
2023-11-23 00:44:20 +01:00
Etan Kissling c33dd2c170
restrict best LC update collection to canonical blocks ()
Simplify best `LightClientUpdate` collection by tracking only canonical
data instead of tracking the best update across all branches within the
sync committee period.

- https://github.com/ethereum/consensus-specs/pull/3553
2023-11-21 23:51:05 +01:00
tersec 7e3aeaea09
automated consensus spec URL updating to v1.4.0-beta.4 () 2023-11-08 05:28:03 +00:00
tersec 556d5e7114
rm unused code () 2023-11-01 05:53:09 +01:00
tersec 62d59daaa7
consensus-spec URL updates to v1.4.0-beta.3 () 2023-10-30 06:44:43 +00:00
tersec 09df3f32b5
add non-SZ getBlobSidecar and BlobSidecar database tests () 2023-10-26 03:40:04 +00:00
tersec 4ddd771127
automated consensus spec URL updating to v1.4.0-beta.3 () 2023-10-19 10:26:38 +00:00
tersec 447786518f
ShufflingRef approach to next-epoch validator duty calculation/prediction ()
* ShufflingRef approach to next-epoch validator duty calculation/prediction

* refactor action_tracker.updateActions to take ShufflingRef + beacon_proposers; refactor maybeUpdateActionTrackerNextEpoch to be separate and reused function; add actual fallback logic

* document one possible set of conditions

* check epoch participation flags and inactivity scores to ensure no penalties and MAX_EFFECTIVE_BALANCE to ensure rewards don't matter

* correctly (un)shuffle each proposer index

* remove debugging assertion
2023-10-10 00:02:07 +00:00
Eugene Kabanov 4fb95d000d
REST server fixes and improvements. ()
* Move from Option[T] to Opt[T] usage.

* Add `finalized` flag.

* Fix compilation issue.

* Http415 error code for some REST API calls.
Introduce more comprehensive error reporting for block calls.
Deprecate decodeEthConsensusVersion() function.

* Bump http-utils.

* Fix copyright year.

* Fix serialization issue.

* Address review comments.

* Post rebase fixes.
2023-09-27 16:45:33 +02:00
tersec 2895a9a05c
automated consensus spec URL updating to v1.4.0-beta.2 () 2023-09-21 18:06:51 +00:00
Etan Kissling e7bc41e005
`blck` --> `forkyBlck` when using `withBlck` / `withStateAndBlck` ()
For symmetry with `forkyState` when using `withState`, and to avoid
problems with shadowing of `blck` when using `withBlck` in `template`,
also rename the injected `blck` to `forkyBlck`.

- https://github.com/nim-lang/Nim/issues/22698
2023-09-21 12:49:14 +02:00
tersec 2b4f987c80
remove pre-v1.4.0 attestation stability subnets ()
* remove pre-v1.4.0 attestation stability subnets

* re-add most of should register stability subnets on attester duties test
2023-09-11 16:03:34 +00:00
tersec 29dbab916c
don't prematurely process blocks waiting for blobs; fix cosmetic head block opt/non-opt logging () 2023-08-27 07:45:24 +00:00
tersec af37a96dbd
don't send fcUs every block if in lc-opt regime and block putatively finalized () 2023-08-15 09:27:56 +00:00
tersec d171303133
update some consensus spec URLs to v1.4.0-beta.1 () 2023-08-12 10:38:06 +00:00
tersec 85e1976ac3
automated consensus spec URL updating to v1.4.0-beta.1 () 2023-08-09 03:58:47 +00:00
tersec 846e7c585b
Revert "Revert "generalize `ShufflingRef` acceleration logic ()" ()" ()
This reverts commit 2ab4592a31.
2023-07-31 13:11:45 +00:00
tersec 2ab4592a31
Revert "generalize `ShufflingRef` acceleration logic ()" ()
This reverts commit eb3a30655b.
2023-07-31 08:05:32 +02:00