* Perform block pre-check before validating execution
When syncing, blocks have not been gossip-validated and are therefore
prone to trivial faults like being known-unviable, duplicate or missing
their parent.
In addition, the duplicate-block check in BlockProcessor was not
considering the quarantine flow and would therefore cause
recently-quarantined blocks to be silenty dropped when their parent
appears delaying the sync end-game and thus causing longer startup
resync time.
This PR verifies trivial conditions before performing execution
validation thus avoiding duplicates and missing parents alike.
It also ensures that the fast-sync EL mode is used for finalized blocks
even if the EL is timing out / slow to respond - this allows the CL to
complete its sync faster and switch to "normal" lock-step at the head of
the chain more quickly, thus also allowing the EL to access the latest
consensensus information earlier.
* oops
* remove unused constant
When the requestmanager is busy fetching blocks, the queue might get
filled with multiple entries of the same root - since there is no
deduplication, requests containing the same root multiple times will be
sent out.
Also, because the items sit in the queue for a long time potentially,
the request might be stale by the time that the manager is ready with
the previous request.
This PR removes the queue and directly fetches the blocks to download
from the quarantine which solves both problems (the quarantine already
de-duplicates and is clean of stale information).
Removing the queue for blobs is left for a future PR.
Co-authored-by: tersec <tersec@users.noreply.github.com>
* early exit `commonAncestor` when comparing with `finalizedHead`
As all `BlockRef` lead to `finalizedHead` (`parent == nil`),
can shortcut in that situation and immediately return `finalizedHead`
if passed as one of the arguments.
* typo in comment
* add test from #5152
Co-authored-by: tersec <tersec@users.noreply.github.com>
* add note about test complexity
* regenerate test summary
---------
Co-authored-by: tersec <tersec@users.noreply.github.com>
With `v1.6.14` there is compilation issue in `trusted_node_sync` where
a type is not inferred automatically anymore for a `nil` instance.
Fix it so we can bump the compiler.
See https://github.com/status-im/nimbus-build-system/pull/63
* Make VC able to understand any type of `/eth/v1/config/spec` response without any changes in source code.
Update compatibility checking.
Now VC is able to obtain any constant from `spec` call.
* Remove RestSpecVC declaration.
* Initial commit.
* Add algorithm in comment.
Remove delays.
Fix logging statement issues.
Change update from epoch to slot.
* Obtain timestamp earlier.
* Add processing delays into algorithm.
* Fix time offset logging to produce integers instead of strings.
* Address review comments.
* Fix copyright year.
Fix updateStatus().
* Remove fields from Slot start log statement.
Fix issues when BN do not support Nimbus Extensions.
Rename metric name and type change.
* Add beacon role to disable time offset check manually.
These tables can't be deleted from (read-only) and would be too slow to
delete from anyway due to the inefficient storage format in use.
* slow down startup clearing too
* remove unused del function
These tables can't be deleted from (read-only) and would be too slow to
delete from anyway due to the inefficient storage format in use.
* slow down startup clearing too
* remove unused del function
The helper function to compute delay until next light client sync task
can be useful from more general purpose contexts. Move to helpers, and
change it to return `Duration` instead of `BeaconTime` for flexibility.
- When syncing `LightClientUpdatesByRange`, and peer replies with
fewer periods than requested, no need to delay next request.
- When `FinalityUpdate` / `OptimisticUpdate` sync fails,
no need to retry immediately.
The `UpdatesByRange` API takes `startPeriod / count`, but is internally
called by `Slice`. Move the logic that converts from the `Slice` to the
caller to reduce complexity inside the used `doRequest` function.
We have several modules that import `nim-eth` for the sole purpose of
its `keys.newRng` function. This function is meanwhile a simple wrapper
around `nim-bearssl`'s `HmacDrbgContext.new()`, so the import doesn't
really serve a use anymore. Replace `keys.newRng` with the direct call
to reduce `nim-eth` imports.
* require sync committee supermajority in CI
To better catch problems with sync committee messages in CI, extend
local testnet simulation to also verify that each block is signed
by a supermajority of the sync committee.
Requires #5083 and #5084
* lint
`produceSyncAggregate` is called in new slot when block is produced,
while the other functions in `sync_committee_msg_pool` are called in
previous slot. So, need to subtract 1 slot when producing sync aggregate
to accept the signatures using the old digest during fork transition.
Override default Json parsing for keystore case objects to avoid
`CaseTransition` logic to be emitted. When parsing the discriminator,
reinitialize the entire object instead of implicitly changing it,
to avoid UB and also possible oversights when extending the object.
See https://github.com/status-im/nim-serialization/pull/59
Before assigning to `genesisData` or returning `cfg`, have to check that
metadata is not `incompatible` to avoid `ProveField` warning.
The way how we use it was already correct (`incompatible` unreachable).
Use `case` syntax to silence the warning, and add comments referring to
the existing checks that make `incompatible` unreachable.
* ensure sync duties for next epoch are registered in time
For attestations, VC queries duties for current and next epoch.
For sync messages, VC queries for current and next period (if soon).
This means that for sync messages we don't actually have the duties for
next epoch in all situations, leading to situation where VC may miss
sync duties in the final slot of an epoch when using. As duties remain
same within a sync committee period, simply copy them over to next epoch
to avoid this situation without adding network latency.
* Update beacon_chain/validator_client/duties_service.nim
Co-authored-by: Jacek Sieka <jacek@status.im>
---------
Co-authored-by: Jacek Sieka <jacek@status.im>
VC currently misses sync committee duties for first slot of most epochs
because the 1 slot offset is not taken into account. Duties for the next
slot must be used during any given current slot. We use the correct slot
for processing the duty, but do not use the correct slot for fetching.
`nim-serialization` is tagged with `{.raises:[SerializationError].}` so
it is no longer sufficient to catch `SszError` in some situations.
`SszError` inherits from `SerializationError`, so broadening the caught
exception types can be done now, to enable bumping `nim-serialization`.
https://github.com/status-im/nimbus-eth2/pull/5043#issuecomment-1584227993#5061 is also needed to bump `nim-serialization`.
Cleanup for `ProveField` warnings in `keystore` module.
Note that `ProveField` is disabled by default in makefile, but sometimes
these pop up when doing a regular `nim c`, and cleaning these may allow
enabling the warning in some future.
* split file loading from parsing in helpers
In `readSszForkedHashedBeaconState` and `readRuntimeConfig`, split the
part that loads the file from the part that parses the file. The parsing
portion can be reused with that, e.g., when loading from the network.
* add missing export marker
* Refactor api.nim to provide more informative failure reasons.
Distinct between unexpected data and unexpected code.
Deprecate Option[T] usage.
* Fix 400 for produceBlindedBlock().
Get proper string conversion for strategy.
* Fix SSZ encoded versions of ProduceBlockResponseV2, ProduceBlockResponseV2 can be received and decoded.
Fix done() warnings.
Bump presto.
* Fix compilation error with new presto.
Use TcpNoDelay option for Web3Signer.
* Fix produceBlockV2() should provide SSZ responses too.
* Address block encoding issue.
* Fix signing test.
* Bump presto.
* Address review comments.
Use separate functions per format when parsing LC data from REST.
This allows to process events from the eventstream more directly,
as they are always JSON not SSZ. And also makes the code cleaner.
Assigning to fields of `var` case objects emits `ProveField` warnings.
We disable them in `make` based builds but they may pop up when building
manually with `nim c`. Suppress them for the `keyGen` function, as we
assign to `result.value` separately from `result.ok` to avoid copying.
* `ProveField` cleanups in `forks`
Some more cleanup for `ProveField` warnings in `forks` module.
Note that `ProveField` is disabled by default in makefile, but sometimes
these pop up when doing a regular `nim c`, and cleaning these may allow
enabling the warning in some future.
* use syntax that works if passed to multiple args of call
When using trusted node sync with `--trusted-block-root`, the remote
server is only trusted for data availability, not for correctness.
As a downloaded genesis state cannot be validated for correctness,
require it to be passed via the network metadata `genesis.ssz` file
for `--trusted-block-root` mode. Network metadata is considered trusted
as it is provided by the user and not by the remote server.
Further adds a check for consistent `genesis_time` when using `StateId`
based trusted node sync. This is just a sanity check to avoid spreading
blatantly incorrect data, similar to existing `genesis_validators_root`
checks.
* Initial commit with both methods enabled: `poll` and `event`.
* Address review comments.
* Address review comments.
Fix copyright years.
* After bump fixes.
Since #3976, CORS functionality is broken. Fix it to work again:
- Use `--rest-allowed-origin` instead of `--keymanager-allowed-origin`
to specify CORS `Access-Control-Allow-Origin` header for beacon-APIs.
- Actually pass CORS config to `nim-presto` once more.
* also pack attestations where LMD vote is orphaned
When `attestation.data.beacon_block_root` gets orphaned, attestations
with a good `attestation.data.target.root` may still be valuable.
The LMD GHOST vote is not relevant for attestation rewards.
Switch to use the FFG vote (`attestation.data.target.root`) instead,
gossip validation ensures it is an ancestor of `beacon_block_root`.
* lint
* allow payload builder client to be function of validator/proposer
* fileExists has side effects on Windows and only Windows
* another not-always-func
* handle one of the `ProveField` warnings
When assigning between `ForkyHashedBeaconState`, suppress `ProveField`
warning, as `tgt.kind == src.kind` was already checked, but compiler
doesn't understand that (as we only `case tgt.kind`).
* Update beacon_chain/spec/forks.nim
* Update beacon_chain/spec/forks.nim
* use correct exception in `parseCmdArg(enr.Record)`
`parseCmdArg` is expected to raise `ValueError` but for `enr.Record` we
currently raise `ConfigurationError`. Change to `ValueError` instead.
* lint
* Refactor api.nim to provide more informative failure reasons.
Distinct between unexpected data and unexpected code.
Deprecate Option[T] usage.
* Fix generated reason to not include opt[t].
* Fix 400 for produceBlindedBlock().
Get proper string conversion for strategy.
* Bump copyright years.
Since #4960, the EL connection status can no longer transition from
`NeverTested` to `Working`. Fix that, and also consider `NeverTested`
connections as online for the purpose of the `el_offline` REST response.
* Clarify addOrphan error/logging
addOrphan returned a bool to indicate success. Change this to a Result
so that different errors can be distinguished.
* Update beacon_chain/consensus_object_pools/block_quarantine.nim
Co-authored-by: tersec <tersec@users.noreply.github.com>
* Update beacon_chain/gossip_processing/gossip_validation.nim
---------
Co-authored-by: tersec <tersec@users.noreply.github.com>
* replace optimisticRoots table with field in BlockRef
* copyright year
* mark finalized blocks as verified on load
* Update beacon_chain/consensus_object_pools/block_dag.nim
Co-authored-by: Etan Kissling <etan@status.im>
* expand non-optimistic block checking to all pre-merge blocks; refactor markBlockVerified to use BlockRef rather than block root and remove superfluous caller in newPayload path replaced by addResolvedHeadBlock BlockRef construction
* don't treat finalized block specially; VALID status is sticky
---------
Co-authored-by: Etan Kissling <etan@status.im>
When doing sync for blocks older than
MIN_EPOCHS_FOR_BLOB_SIDECARS_REQUESTS, we skip the blobs by range
request, but we then pass en empty blob sequence to
validation, which then fails.
To fix this: Use an Option[Blobsidecars] to allow expressing the
distinction between "empty blob sequence" and "blobs unavailable". Use
the latter for "old" blocks, and don't attempt to run blob validation.
`SyncCommitteeMsgPool` grouped messages by their `beacon_block_root`.
This is problematic around sync committee period boundaries and forks.
Around sync committee period boundaries, members from both the current
and next sync committee may sign the same `beacon_block_root`; mixing
the signatures from both committees together is a mistake. Likewise,
around fork transitions, the `signing_root` changes, so those messages
also need to be segregated.
The validator beacon APIs `getAttesterDuties`, `getProposerDuties`, and
`getSyncCommitteeDuties`, have reported the `execution_optimistic`
state for the current head block. This can lead to a race if duties are
requested around the slot start, if a new head block is currently being
processed by the EL, during which the BN head may be briefly optimistic.
`execution_optimistic` is documented in beacon APIs as:
> True if the response references an unverified execution payload.
> Optimistic information may be invalidated at a later time.
> If the field is not present, assume the False value.
As the duty endpoints reference the shuffling dependent root instead of
the currently selected head block, `execution_optimistic` is now fetched
based on that shuffling dependent block root. As this dependent block is
in the past it doesn't usually become optimistic when adding new blocks.
Note that the endpoints requested 4/8 seconds into the slot that perform
the actual duties instead of just querying for duty schedule, still
report `execution_optimistic` based on the BN head block.
When fetching historical `getSyncCommitteeDuties` for the very first
sync committee period, the case must be handled where Altair may not
have been scheduled on a sync committee period boundary.
When an uncached `ShufflingRef` is requested, we currently replay state
which can take several seconds. Acceleration is possible by:
1. Start from any state with locked-in `get_active_validator_indices`.
Any blocks / slots applied to such a state can only affect that
result for future epochs, so are viable for querying target epoch.
`compute_activation_exit_epoch(state.slot.epoch) > target.epoch`
2. Determine highest common ancestor among `state` and `target.blck`.
At the ancestor slot, same rules re `get_active_validator_indices`.
`compute_activation_exit_epoch(ancestorSlot.epoch) > target.epoch`
3. We now have a `state` that shares history with `target.blck` up
through a common ancestor slot. Any blocks / slots that the `state`
contains, which are not part of the `target.blck` history, affect
`get_active_validator_indices` at epochs _after_ `target.epoch`.
4. Select `state.randao_mixes[N]` that is closest to common ancestor.
Either direction is fine (above / below ancestor).
5. From that RANDAO mix, mix in / out all RANDAO reveals from blocks
in-between. This is just an XOR operation, so fully reversible.
`mix = mix xor SHA256(blck.message.body.randao_reveal)`
6. Compute the attester dependent slot from `target.epoch`.
`if epoch >= 2: (target.epoch - 1).start_slot - 1 else: GENESIS_SLOT`
7. Trace back from `target.blck` to the attester dependent slot.
We now have the destination for which we want to obtain RANDAO.
8. Mix in all RANDAO reveals from blocks up through the `dependentBlck`.
Same method, no special handling necessary for epoch transitions.
9. Combine `get_active_validator_indices` from `state` at `target.epoch`
with the recovered RANDAO value at `dependentBlck` to obtain the
requested shuffling, and construct the `ShufflingRef` without replay.
* more tests and simplify logic
* test with different number of deposits per branch
* Update beacon_chain/consensus_object_pools/blockchain_dag.nim
Co-authored-by: Jacek Sieka <jacek@status.im>
* `commonAncestor` tests
* lint
---------
Co-authored-by: Jacek Sieka <jacek@status.im>
* fix SSZ response for `produceBlindedBlock`
In `produceBlindedBlock`, we sent the `ForkedBlindedBeaconBlock` when
requested to reply in SSZ format. However, expected result is just the
inner `ForkyBlindedBeaconBlock` together with `eth-consensus-version`.
Note: We do not use SSZ format in our VC for this endpoint at this time,
which explains why we haven't noticed earlier.
* fix Altair/Phase0
* Incremental pruning
When turning on pruning the first time the current pruning algorithm
will prune the full database at startup. This delays restart
unnecessarily, since all of the pruned space is not needed at once.
This PR introduces incremental pruning such that we will never prune
more than 32 blocks or the sync speed, whichever is higher.
This mode is expected to become default in a follow-up release.
* Kzg: Load trusted setup
* scripts/launch_local_testnet.sh: set FIELD_ELEMENTS_PER_BLOB
* Use right setup file for mainnet/minimal
* Force rebuild
* Add comment explaining why build with -f
`attachMerkleProofs` is used by `mockUpdateStateForNewDeposit` to create
a single deposit. The function doesn't work correctly when trying with
with multiple deposits, though. Fix this to enable more complex tests,
and also return the `deposit_root` for forming matching `Eth1Data`.
* final portion of non-trivial v1.3.0 bumps
Updates unchanged logic to latest v1.3.0 consensus-specs refs,
and cleans up surrounding sections / syncs comments, and so on.
```
https://github.com/ethereum/consensus-specs/(blob|tree)/(?!v1\.3\.0/)
```
* lint
* linebreak
* cleanup `state_transition_epoch` and bump to v1.3.0
More v1.3.0 consensus-specs bumps, focused on `state_transition_epoch`.
Also fixed `current_epoch` spurious style check warning, and cleanup.
* Update beacon_chain/spec/state_transition_epoch.nim
Make intent clearer about when to expect `bls_to_execution_changes`,
and replace test-time errors with compile-time errors if the field
is renamed or goes away in the future.
The `SignedContributionAndProof: invalid contribution signature` check
is sometimes hit around fork boundaries when running local testnet.
To avoid failing CI, revert this isntance to a plain `errReject` until
the underlying problem is addressed.
We already updated the field order in the actual `ExecutionPayload`,
but in init code and tests / logs etc we still used the old order.
Update those occurrences to also match the field order in the struct.
Furthermore, add `excess_data_gas` to last entry in `test_eth1_monitor`.
Updates gossip validation spec references to v1.3.0 and fixes an
incorrect reference to "signed_aggregate_and_proof" in sync contribution
documentation.
The `SAFE_SLOTS_TO_UPDATE_JUSTIFIED` constant is no longer used as the
bouncing attack fix was removed:
https://github.com/ethereum/consensus-specs/pull/3290
Note: Some test networks still define the constant, ignoring the config
constant for now until it is no longer used.
Only comment changes:
- Bump refs to final v1.3.0 spec
- Align documentation style in various `BeaconState` structures
- Add `justification_bits` / `historical_roots` comment from spec
- Remove `previous_justified_checkpoint` from non-phase0 (same as spec)
- Cleanup some `Modified` tags
Jacek Sieka
henridf
tersec
Etan Kissling
Eugene Kabanov
Zahary Karadjov
Tanguy
Jacob Shufro
jangko