status-im
/
nimbus-eth2
mirror of https://github.com/status-im/nimbus-eth2.git
2
0
Fork 0
Code Issues Projects Releases Wiki Activity
nimbus-eth2/tests/test_light_client.nim

191 lines
7.2 KiB
Nim
Raw Normal View History

add option to collect light client data (#3474) Light clients require full nodes to serve additional data so that they can stay in sync with the network. This patch adds a new launch option `--import-light-client-data` to configure what data to make available. For now, data is only kept in memory; it is not persisted at this time. Note that data is only locally collected, a separate patch is needed to actually make it availble over the network. `--serve-light-client-data` will be used for serving data, but is not functional yet outside tests.
2022-03-11 20:28:10 +00:00
# beacon_chain
# Copyright (c) 2021-2022 Status Research & Development GmbH
# Licensed and distributed under either of
# * MIT license (license terms in the root directory or at https://opensource.org/licenses/MIT).
# * Apache v2 license (license terms in the root directory or at https://www.apache.org/licenses/LICENSE-2.0).
# at your option. This file may not be copied, modified, or distributed except according to those terms.
{.used.}
# This implements the pre-release proposal of the libp2p based light client sync
# protocol. See https://github.com/ethereum/consensus-specs/pull/2802
import
# Status libraries
chronicles, eth/keys, stew/objects, taskpools,
# Beacon chain internals
../beacon_chain/consensus_object_pools/
[block_clearance, block_quarantine, blockchain_dag],
../beacon_chain/spec/[forks, helpers, light_client_sync, state_transition],
# Test utilities
./testutil, ./testdbutil
suite "Light client" & preset():
let
cfg = block:
var res = defaultRuntimeConfig
res.ALTAIR_FORK_EPOCH = GENESIS_EPOCH + 1
res
altairStartSlot = cfg.ALTAIR_FORK_EPOCH.start_slot
proc advanceToSlot(
dag: ChainDAGRef,
targetSlot: Slot,
verifier: var BatchVerifier,
quarantine: var Quarantine,
attested = true,
syncCommitteeRatio = 0.75) =
var cache: StateCache
const maxAttestedSlotsPerPeriod = 3 * SLOTS_PER_EPOCH
while true:
remove StateData (#3507) One more step on the journey to reduce `BlockRef` usage across the codebase - this one gets rid of `StateData` whose job was to keep track of which block was last assigned to a state - these duties have now been taken over by `latest_block_root`, a fairly recent addition that computes this block root from state data (at a small cost that should be insignificant) 99% mechanical change.
2022-03-16 07:20:40 +00:00
var slot = getStateField(dag.headState, slot)
add option to collect light client data (#3474) Light clients require full nodes to serve additional data so that they can stay in sync with the network. This patch adds a new launch option `--import-light-client-data` to configure what data to make available. For now, data is only kept in memory; it is not persisted at this time. Note that data is only locally collected, a separate patch is needed to actually make it availble over the network. `--serve-light-client-data` will be used for serving data, but is not functional yet outside tests.
2022-03-11 20:28:10 +00:00
doAssert targetSlot >= slot
if targetSlot == slot: break
# When there is a large jump, skip to the end of the current period,
# create blocks for a few epochs to finalize it, then proceed
let
nextPeriod = slot.sync_committee_period + 1
periodEpoch = nextPeriod.start_epoch
periodSlot = periodEpoch.start_slot
checkpointSlot = periodSlot - maxAttestedSlotsPerPeriod
if targetSlot > checkpointSlot and checkpointSlot > dag.head.slot:
var info: ForkedEpochInfo
remove StateData (#3507) One more step on the journey to reduce `BlockRef` usage across the codebase - this one gets rid of `StateData` whose job was to keep track of which block was last assigned to a state - these duties have now been taken over by `latest_block_root`, a fairly recent addition that computes this block root from state data (at a small cost that should be insignificant) 99% mechanical change.
2022-03-16 07:20:40 +00:00
doAssert process_slots(cfg, dag.headState, checkpointSlot,
add option to collect light client data (#3474) Light clients require full nodes to serve additional data so that they can stay in sync with the network. This patch adds a new launch option `--import-light-client-data` to configure what data to make available. For now, data is only kept in memory; it is not persisted at this time. Note that data is only locally collected, a separate patch is needed to actually make it availble over the network. `--serve-light-client-data` will be used for serving data, but is not functional yet outside tests.
2022-03-11 20:28:10 +00:00
cache, info, flags = {}).isOk()
slot = checkpointSlot
# Create blocks for final few epochs
let blocks = min(targetSlot - slot, maxAttestedSlotsPerPeriod)
remove StateData (#3507) One more step on the journey to reduce `BlockRef` usage across the codebase - this one gets rid of `StateData` whose job was to keep track of which block was last assigned to a state - these duties have now been taken over by `latest_block_root`, a fairly recent addition that computes this block root from state data (at a small cost that should be insignificant) 99% mechanical change.
2022-03-16 07:20:40 +00:00
for blck in makeTestBlocks(dag.headState, cache, blocks.int,
add option to collect light client data (#3474) Light clients require full nodes to serve additional data so that they can stay in sync with the network. This patch adds a new launch option `--import-light-client-data` to configure what data to make available. For now, data is only kept in memory; it is not persisted at this time. Note that data is only locally collected, a separate patch is needed to actually make it availble over the network. `--serve-light-client-data` will be used for serving data, but is not functional yet outside tests.
2022-03-11 20:28:10 +00:00
attested, syncCommitteeRatio, cfg):
let added =
case blck.kind
of BeaconBlockFork.Phase0:
const nilCallback = OnPhase0BlockAdded(nil)
dag.addHeadBlock(verifier, blck.phase0Data, nilCallback)
of BeaconBlockFork.Altair:
const nilCallback = OnAltairBlockAdded(nil)
dag.addHeadBlock(verifier, blck.altairData, nilCallback)
of BeaconBlockFork.Bellatrix:
const nilCallback = OnBellatrixBlockAdded(nil)
dag.addHeadBlock(verifier, blck.bellatrixData, nilCallback)
check: added.isOk()
dag.updateHead(added[], quarantine)
setup:
const num_validators = SLOTS_PER_EPOCH
let
validatorMonitor = newClone(ValidatorMonitor.init())
dag = ChainDAGRef.init(
cfg, makeTestDB(num_validators), validatorMonitor, {},
serveLightClientData = true,
importLightClientData = ImportLightClientData.OnlyNew)
quarantine = newClone(Quarantine.init())
enable `styleCheck:usages` (#3573) Some upstream repos still need fixes, but this gets us close enough that style hints can be enabled by default. In general, "canonical" spellings are preferred even if they violate nep-1 - this applies in particular to spec-related stuff like `genesis_validators_root` which appears throughout the codebase.
2022-04-08 16:22:49 +00:00
taskpool = Taskpool.new()
add option to collect light client data (#3474) Light clients require full nodes to serve additional data so that they can stay in sync with the network. This patch adds a new launch option `--import-light-client-data` to configure what data to make available. For now, data is only kept in memory; it is not persisted at this time. Note that data is only locally collected, a separate patch is needed to actually make it availble over the network. `--serve-light-client-data` will be used for serving data, but is not functional yet outside tests.
2022-03-11 20:28:10 +00:00
var verifier = BatchVerifier(rng: keys.newRng(), taskpool: taskpool)
test "Pre-Altair":
# Genesis
check:
remove StateData (#3507) One more step on the journey to reduce `BlockRef` usage across the codebase - this one gets rid of `StateData` whose job was to keep track of which block was last assigned to a state - these duties have now been taken over by `latest_block_root`, a fairly recent addition that computes this block root from state data (at a small cost that should be insignificant) 99% mechanical change.
2022-03-16 07:20:40 +00:00
dag.headState.kind == BeaconStateFork.Phase0
add option to collect light client data (#3474) Light clients require full nodes to serve additional data so that they can stay in sync with the network. This patch adds a new launch option `--import-light-client-data` to configure what data to make available. For now, data is only kept in memory; it is not persisted at this time. Note that data is only locally collected, a separate patch is needed to actually make it availble over the network. `--serve-light-client-data` will be used for serving data, but is not functional yet outside tests.
2022-03-11 20:28:10 +00:00
dag.getBestLightClientUpdateForPeriod(0.SyncCommitteePeriod).isNone
dag.getLatestLightClientUpdate.isNone
# Advance to last slot before Altair
dag.advanceToSlot(altairStartSlot - 1, verifier, quarantine[])
check:
remove StateData (#3507) One more step on the journey to reduce `BlockRef` usage across the codebase - this one gets rid of `StateData` whose job was to keep track of which block was last assigned to a state - these duties have now been taken over by `latest_block_root`, a fairly recent addition that computes this block root from state data (at a small cost that should be insignificant) 99% mechanical change.
2022-03-16 07:20:40 +00:00
dag.headState.kind == BeaconStateFork.Phase0
add option to collect light client data (#3474) Light clients require full nodes to serve additional data so that they can stay in sync with the network. This patch adds a new launch option `--import-light-client-data` to configure what data to make available. For now, data is only kept in memory; it is not persisted at this time. Note that data is only locally collected, a separate patch is needed to actually make it availble over the network. `--serve-light-client-data` will be used for serving data, but is not functional yet outside tests.
2022-03-11 20:28:10 +00:00
dag.getBestLightClientUpdateForPeriod(0.SyncCommitteePeriod).isNone
dag.getLatestLightClientUpdate.isNone
# Advance to Altair
dag.advanceToSlot(altairStartSlot, verifier, quarantine[])
check:
remove StateData (#3507) One more step on the journey to reduce `BlockRef` usage across the codebase - this one gets rid of `StateData` whose job was to keep track of which block was last assigned to a state - these duties have now been taken over by `latest_block_root`, a fairly recent addition that computes this block root from state data (at a small cost that should be insignificant) 99% mechanical change.
2022-03-16 07:20:40 +00:00
dag.headState.kind == BeaconStateFork.Altair
add option to collect light client data (#3474) Light clients require full nodes to serve additional data so that they can stay in sync with the network. This patch adds a new launch option `--import-light-client-data` to configure what data to make available. For now, data is only kept in memory; it is not persisted at this time. Note that data is only locally collected, a separate patch is needed to actually make it availble over the network. `--serve-light-client-data` will be used for serving data, but is not functional yet outside tests.
2022-03-11 20:28:10 +00:00
dag.getBestLightClientUpdateForPeriod(0.SyncCommitteePeriod).isNone
dag.getLatestLightClientUpdate.isNone
test "Light client sync":
# Advance to Altair
dag.advanceToSlot(altairStartSlot, verifier, quarantine[])
# Track trusted checkpoint for light client
let
enable `styleCheck:usages` (#3573) Some upstream repos still need fixes, but this gets us close enough that style hints can be enabled by default. In general, "canonical" spellings are preferred even if they violate nep-1 - this applies in particular to spec-related stuff like `genesis_validators_root` which appears throughout the codebase.
2022-04-08 16:22:49 +00:00
genesis_validators_root = dag.genesis_validators_root
remove StateData (#3507) One more step on the journey to reduce `BlockRef` usage across the codebase - this one gets rid of `StateData` whose job was to keep track of which block was last assigned to a state - these duties have now been taken over by `latest_block_root`, a fairly recent addition that computes this block root from state data (at a small cost that should be insignificant) 99% mechanical change.
2022-03-16 07:20:40 +00:00
trusted_block_root = dag.head.root
add option to collect light client data (#3474) Light clients require full nodes to serve additional data so that they can stay in sync with the network. This patch adds a new launch option `--import-light-client-data` to configure what data to make available. For now, data is only kept in memory; it is not persisted at this time. Note that data is only locally collected, a separate patch is needed to actually make it availble over the network. `--serve-light-client-data` will be used for serving data, but is not functional yet outside tests.
2022-03-11 20:28:10 +00:00
# Advance to target slot
const
headPeriod = 2.SyncCommitteePeriod
periodEpoch = headPeriod.start_epoch
headSlot = (periodEpoch + 2).start_slot + 5
dag.advanceToSlot(headSlot, verifier, quarantine[])
remove StateData (#3507) One more step on the journey to reduce `BlockRef` usage across the codebase - this one gets rid of `StateData` whose job was to keep track of which block was last assigned to a state - these duties have now been taken over by `latest_block_root`, a fairly recent addition that computes this block root from state data (at a small cost that should be insignificant) 99% mechanical change.
2022-03-16 07:20:40 +00:00
let currentSlot = getStateField(dag.headState, slot)
add option to collect light client data (#3474) Light clients require full nodes to serve additional data so that they can stay in sync with the network. This patch adds a new launch option `--import-light-client-data` to configure what data to make available. For now, data is only kept in memory; it is not persisted at this time. Note that data is only locally collected, a separate patch is needed to actually make it availble over the network. `--serve-light-client-data` will be used for serving data, but is not functional yet outside tests.
2022-03-11 20:28:10 +00:00
# Initialize light client store
let bootstrap = dag.getLightClientBootstrap(trusted_block_root)
update light client server for DAG failure modes (#3514) Gracefully handles the new failure modes recently introduced to the DAG as part of https://github.com/status-im/nimbus-eth2/pull/3513 Data that is deemed to exist but fails to load leads to an error log to avoid suppressing logic errors accidentally. In `verifyFinalization` mode, the assertions remain active.
2022-03-20 10:58:59 +00:00
check bootstrap.isOk
add option to collect light client data (#3474) Light clients require full nodes to serve additional data so that they can stay in sync with the network. This patch adds a new launch option `--import-light-client-data` to configure what data to make available. For now, data is only kept in memory; it is not persisted at this time. Note that data is only locally collected, a separate patch is needed to actually make it availble over the network. `--serve-light-client-data` will be used for serving data, but is not functional yet outside tests.
2022-03-11 20:28:10 +00:00
var storeRes = initialize_light_client_store(
trusted_block_root, bootstrap.get)
error and progress codes for light client sync (#3490) When syncing as a light client, different behaviour is needed to handle the various ways how errors may occur. The existing logic for blocks can also be applied to light client objects: - `Invalid`: Malformed object that is clearly an error by its producer. - `MissingParent`: More data is needed to decide applicability. - `UnviableFork`: Object may be valid but will never apply on this fork. - `Duplicate`: No errors were encountered but the object was not useful.
2022-03-14 09:25:54 +00:00
check storeRes.isOk
add option to collect light client data (#3474) Light clients require full nodes to serve additional data so that they can stay in sync with the network. This patch adds a new launch option `--import-light-client-data` to configure what data to make available. For now, data is only kept in memory; it is not persisted at this time. Note that data is only locally collected, a separate patch is needed to actually make it availble over the network. `--serve-light-client-data` will be used for serving data, but is not functional yet outside tests.
2022-03-11 20:28:10 +00:00
template store(): auto = storeRes.get
# Sync to latest sync committee period
var numIterations = 0
while store.finalized_header.slot.sync_committee_period + 1 < headPeriod:
let
period =
if store.next_sync_committee.isZeroMemory:
store.finalized_header.slot.sync_committee_period
else:
store.finalized_header.slot.sync_committee_period + 1
bestUpdate = dag.getBestLightClientUpdateForPeriod(period)
res = process_light_client_update(
store, bestUpdate.get, currentSlot, cfg, genesis_validators_root)
check:
bestUpdate.isSome
bestUpdate.get.finalized_header.slot.sync_committee_period == period
error and progress codes for light client sync (#3490) When syncing as a light client, different behaviour is needed to handle the various ways how errors may occur. The existing logic for blocks can also be applied to light client objects: - `Invalid`: Malformed object that is clearly an error by its producer. - `MissingParent`: More data is needed to decide applicability. - `UnviableFork`: Object may be valid but will never apply on this fork. - `Duplicate`: No errors were encountered but the object was not useful.
2022-03-14 09:25:54 +00:00
res.isOk
add option to collect light client data (#3474) Light clients require full nodes to serve additional data so that they can stay in sync with the network. This patch adds a new launch option `--import-light-client-data` to configure what data to make available. For now, data is only kept in memory; it is not persisted at this time. Note that data is only locally collected, a separate patch is needed to actually make it availble over the network. `--serve-light-client-data` will be used for serving data, but is not functional yet outside tests.
2022-03-11 20:28:10 +00:00
store.finalized_header == bestUpdate.get.finalized_header
inc numIterations
if numIterations > 20: doAssert false # Avoid endless loop on test failure
# Sync to latest update
let
latestUpdate = dag.getLatestLightClientUpdate
res = process_light_client_update(
store, latestUpdate.get, currentSlot, cfg, genesis_validators_root)
check:
latestUpdate.isSome
remove StateData (#3507) One more step on the journey to reduce `BlockRef` usage across the codebase - this one gets rid of `StateData` whose job was to keep track of which block was last assigned to a state - these duties have now been taken over by `latest_block_root`, a fairly recent addition that computes this block root from state data (at a small cost that should be insignificant) 99% mechanical change.
2022-03-16 07:20:40 +00:00
latestUpdate.get.attested_header.slot == dag.head.parent.slot
error and progress codes for light client sync (#3490) When syncing as a light client, different behaviour is needed to handle the various ways how errors may occur. The existing logic for blocks can also be applied to light client objects: - `Invalid`: Malformed object that is clearly an error by its producer. - `MissingParent`: More data is needed to decide applicability. - `UnviableFork`: Object may be valid but will never apply on this fork. - `Duplicate`: No errors were encountered but the object was not useful.
2022-03-14 09:25:54 +00:00
res.isOk
add option to collect light client data (#3474) Light clients require full nodes to serve additional data so that they can stay in sync with the network. This patch adds a new launch option `--import-light-client-data` to configure what data to make available. For now, data is only kept in memory; it is not persisted at this time. Note that data is only locally collected, a separate patch is needed to actually make it availble over the network. `--serve-light-client-data` will be used for serving data, but is not functional yet outside tests.
2022-03-11 20:28:10 +00:00
store.finalized_header == latestUpdate.get.finalized_header
store.optimistic_header == latestUpdate.get.attested_header
test "Init from checkpoint":
# Fetch genesis state
remove StateData (#3507) One more step on the journey to reduce `BlockRef` usage across the codebase - this one gets rid of `StateData` whose job was to keep track of which block was last assigned to a state - these duties have now been taken over by `latest_block_root`, a fairly recent addition that computes this block root from state data (at a small cost that should be insignificant) 99% mechanical change.
2022-03-16 07:20:40 +00:00
let genesisState = assignClone dag.headState
add option to collect light client data (#3474) Light clients require full nodes to serve additional data so that they can stay in sync with the network. This patch adds a new launch option `--import-light-client-data` to configure what data to make available. For now, data is only kept in memory; it is not persisted at this time. Note that data is only locally collected, a separate patch is needed to actually make it availble over the network. `--serve-light-client-data` will be used for serving data, but is not functional yet outside tests.
2022-03-11 20:28:10 +00:00
# Advance to target slot for checkpoint
let finalizedSlot =
((altairStartSlot.sync_committee_period + 1).start_epoch + 2).start_slot
dag.advanceToSlot(finalizedSlot, verifier, quarantine[])
# Initialize new DAG from checkpoint
let cpDb = BeaconChainDB.new("", inMemory = true)
ChainDAGRef.preInit(
stylecheck fixes (#3595)
2022-04-15 12:46:56 +00:00
cpDb, genesisState[],
remove StateData (#3507) One more step on the journey to reduce `BlockRef` usage across the codebase - this one gets rid of `StateData` whose job was to keep track of which block was last assigned to a state - these duties have now been taken over by `latest_block_root`, a fairly recent addition that computes this block root from state data (at a small cost that should be insignificant) 99% mechanical change.
2022-03-16 07:20:40 +00:00
dag.headState, dag.getForkedBlock(dag.head.bid).get)
add option to collect light client data (#3474) Light clients require full nodes to serve additional data so that they can stay in sync with the network. This patch adds a new launch option `--import-light-client-data` to configure what data to make available. For now, data is only kept in memory; it is not persisted at this time. Note that data is only locally collected, a separate patch is needed to actually make it availble over the network. `--serve-light-client-data` will be used for serving data, but is not functional yet outside tests.
2022-03-11 20:28:10 +00:00
let cpDag = ChainDAGRef.init(
cfg, cpDb, validatorMonitor, {},
serveLightClientData = true,
importLightClientData = ImportLightClientData.Full)
# Advance by a couple epochs
for i in 1'u64 .. 10:
let headSlot = (finalizedSlot.epoch + i).start_slot
cpDag.advanceToSlot(headSlot, verifier, quarantine[])
check true