Commit Graph

52 Commits

Author SHA1 Message Date
tersec a07d14cd99
remove unused imports in tests/ (#3713) 2022-06-07 17:05:06 +00:00
Jacek Sieka 48f01186d6
fix unnecessary HashList/HashArray cache invalidation (#3660)
* SSZ `[]` -> `mitem`
* `[]` -> `item`

immutable access via mutable instance cannot rely on template
overloading, and `[]` cannot be a `func` because of special seq handling
in compiler.
2022-05-30 13:30:42 +00:00
tersec 84588b34da
var => let in specs/ and tests/ (#3425) 2022-02-20 20:13:06 +00:00
Jacek Sieka 805e85e1ff
time: spring cleaning (#3262)
Time in the beacon chain is expressed relative to the genesis time -
this PR creates a `beacon_time` module that collects helpers and
utilities for dealing the time units - the new module does not deal with
actual wall time (that's remains in `beacon_clock`).

Collecting the time related stuff in one place makes it easier to find,
avoids some circular imports and allows more easily identifying the code
actually needs wall time to operate.

* move genesis-time-related functionality into `spec/beacon_time`
* avoid using `chronos.Duration` for time differences - it does not
support negative values (such as when something happens earlier than it
should)
* saturate conversions between `FAR_FUTURE_XXX`, so as to avoid
overflows
* fix delay reporting in validator client so it uses the expected
deadline of the slot, not "closest wall slot"
* simplify looping over the slots of an epoch
* `compute_start_slot_at_epoch` -> `start_slot`
* `compute_epoch_at_slot` -> `epoch`

A follow-up PR will (likely) introduce saturating arithmetic for the
time units - this is merely code moves, renames and fixing of small
bugs.
2022-01-11 11:01:54 +01:00
Jacek Sieka 20e700fae4
Harden CommitteeIndex, SubnetId, SyncSubcommitteeIndex (#3259)
* Harden CommitteeIndex, SubnetId, SyncSubcommitteeIndex

Harden the use of `CommitteeIndex` et al to prevent future issues by
using a distinct type, then validating before use in several cases -
datatypes in spec are kept simple though so that invalid data still can
be read.

* fix invalid epoch used in REST
`/eth/v1/beacon/states/{state_id}/committees` committee length (could
return invalid data)
* normalize some variable names
* normalize committee index loops
* fix `RestAttesterDuty` to use `uint64` for `validator_committee_index`
* validate `CommitteeIndex` on ingress in REST API
* update rest rules with stricter parsing
* better REST serializers
* save lots of memory by not using `zip` ...at least a few bytes!
2022-01-09 01:28:49 +02:00
Jacek Sieka 0e2b4e39fa
REST JSON support improvements (#3232)
* support downloading blocks / states via JSON in addition to SSZ -
slow, but needed for infura support - SSZ is still used when server
supports it
* use common forked block/state reader in REST API
* fix stack overflows in REST JSON decoder
* fix invalid serialization of `justification_bits` in
`/eth/v1/debug/beacon/states` and `/eth/v2/debug/beacon/states`
* fix REST client to use `/eth/...` instead of `/api/eth/...`, update
"default" urls to expose REST api via `/eth` as well as this is what the
standard says - `/api` was added early on based on an example "base url"
in the spec that has been removed since
* expose Nimbus REST extensions via `/nimbus` in addition to
`/api/nimbus` to stay consistent with `/eth`
* fix invalid state root when reading states via REST
* fix recursive imports in `spec/ssz_codec`
* remove usages of `serialization.useCustomSerialization` - fickle
2022-01-06 08:38:40 +01:00
Jacek Sieka a223d62b07
Cleanups (#3123)
Renames and cleanups split out from the validator monitoring branch, so
as to reduce conflict area vs other PR:s

* add constants for expected message timing
* name validators after the messages they validate, mostly, to make
grepping easier
* unify field naming of EpochInfo across forks to make cross-fork code
easier
2021-11-25 13:20:36 +01:00
Jacek Sieka c40cc6cec1 clean up fork enum and field names
* single naming strategy
* simplify some fork code
* simplify forked block production
2021-10-19 11:06:38 +03:00
Jacek Sieka f90b2b8b1f
reward accounting for altair+ (#2981)
Similar to the existing `RewardInfo`, this PR adds the infrastructure
needed to export epoch processing information from altair+. Because
accounting is done somewhat differently, the PR uses a fork-specific
object to extrct the information in order to make the cost on the spec
side low.

* RewardInfo -> EpochInfo, ForkedEpochInfo
* use array for computing new sync committee
* avoid repeated total active balance computations in block processing
* simplify proposer index check
* simplify epoch transition tests
* pre-compute base increment and reuse in epoch processing, and a few
other small optimizations

This PR introduces the type and does the heavy lifting in terms of
refactoring - the tools that use the accounting will need separate PR:s
(as well as refinements to the exportred information)
2021-10-13 16:24:36 +02:00
Etan Kissling 7bbe0258d0
allow testing genesis with Altair (#2866)
The mocking framework was limited to chain configurations that do not
include Altair at genesis. This patch extends it so that genesis states
can be generated that are already upgraded to Altair. This is useful for
tests such as eth2spec/test/altair/unittests/test_sync_protocol.py.
2021-09-13 22:56:37 +00:00
tersec 43a976f89b
proc -> func in ncli/, research/, and test/ (#2818) 2021-08-25 14:51:52 +00:00
Jacek Sieka 7a622e8505
rework spec imports (#2779)
The spec imports are a mess to work with, so this branch cleans them up
a bit to ensure that we avoid generic sandwitches and that importing
stuff generally becomes easier.

* reexport crypto/digest/presets because these are part of the public
symbol set of the rest of the spec types
* don't export `merge` types from `base` - this causes circular deps
* fix circular deps in `ssz/spec_types` - this is the first step in
disentangling ssz from spec
* be explicit about phase0 vs altair - longer term, `altair` will become
the "natural" type set, then merge and so on, so no point in giving
`phase0` special preferential treatment
2021-08-12 13:08:20 +00:00
Jacek Sieka 9697b73e71
forkedbeaconstate_helpers -> forks (#2772)
Simpler module name for stuff that covers forks

* check that runtime config matches database state
* also include some assorted altair cleanups
* use "standard" genesis fork in local testnet to work around missing
runtime config support
2021-08-10 22:46:35 +02:00
Jacek Sieka 23eea197f6
Implement split preset/config support (#2710)
* Implement split preset/config support

This is the initial bulk refactor to introduce runtime config values in
a number of places, somewhat replacing the existing mechanism of loading
network metadata.

It still needs more work, this is the initial refactor that introduces
runtime configuration in some of the places that need it.

The PR changes the way presets and constants work, to match the spec. In
particular, a "preset" now refers to the compile-time configuration
while a "cfg" or "RuntimeConfig" is the dynamic part.

A single binary can support either mainnet or minimal, but not both.
Support for other presets has been removed completely (can be readded,
in case there's need).

There's a number of outstanding tasks:

* `SECONDS_PER_SLOT` still needs fixing
* loading custom runtime configs needs redoing
* checking constants against YAML file

* yeerongpilly support

`build/nimbus_beacon_node --network=yeerongpilly --discv5:no --log-level=DEBUG`

* load fork epoch from config

* fix fork digest sent in status
* nicer error string for request failures
* fix tools

* one more

* fixup

* fixup

* fixup

* use "standard" network definition folder in local testnet

Files are loaded from their standard locations, including genesis etc,
to conform to the format used in the `eth2-networks` repo.

* fix launch scripts, allow unknown config values

* fix base config of rest test

* cleanups

* bundle mainnet config using common loader
* fix spec links and names
* only include supported preset in binary

* drop yeerongpilly, add altair-devnet-0, support boot_enr.yaml
2021-07-12 15:01:38 +02:00
tersec b1d5609171
remove false OnBlockAdded dependency on phase0 HashedBeaconState (#2661)
* remove false OnBlockAdded dependency on phase.HashedBeaconState

* introduce altair data types into block_clearance; update some alpha.6 spec refs to alpha.7; add get_active_validator_indices_len ForkedHashedBeaconState wrapper

* switch many modules from using datatypes (with phase0 states/blocks) to datatypes/base (fork-independent); update spec refs from alpha.6 to alpha.7 and remove rm'd G2_POINT_AT_INFINITY

* switch more modules from using datatypes (with phase0 states/blocks) to datatypes/base (fork-independent); update spec refs from alpha.6 to alpha.7

* remove unnecessary phase0-only wrapper of get_attesting_indices(); allow signatures_batch to process either fork; remove O(n^2) nested loop in process_inactivity_updates(); add altair support to getAttestationsforTestBlock()

* add Altair versions of asSigVerified(), asTrusted(), and makeBeaconBlock()

* fix spec URL to be Altair for Altair makeBeaconBlock()
2021-06-21 08:35:24 +00:00
tersec 146fa48454
use ForkedHashedBeaconState in StateData (#2634)
* use ForkedHashedBeaconState in StateData

* fix FAR_FUTURE_EPOCH -> slot overflow; almost always use assign()

* avoid stack allocation in maybeUpgradeStateToAltair()

* create and use dispatch functions for check_attester_slashing(), check_proposer_slashing(), and check_voluntary_exit()

* use getStateRoot() instead of various state.data.hbsPhase0.root

* remove withStateVars.hashedState(), which doesn't work as a design anymore

* introduce spec/datatypes/altair into beacon_chain_db

* fix inefficient codegen for getStateField(largeStateField)

* state_transition_slots() doesn't either need/use blocks or runtime presets

* combine process_slots(HBS)/state_transition_slots(HBS) which differ only in last-slot htr optimization

* getStateField(StateData, ...) was replaced by getStateField(ForkedHashedBeaconState, ...)

* fix rollback

* switch some state_transition(), process_slots, makeTestBlocks(), etc to use ForkedHashedBeaconState

* remove state_transition(phase0.HashedBeaconState)

* remove process_slots(phase0.HashedBeaconState)

* remove state_transition_block(phase0.HashedBeaconState)

* remove unused callWithBS(); separate case expression from if statement

* switch back from nested-ref-object construction to (ref Foo)(Bar())
2021-06-11 20:51:46 +03:00
tersec c06ffc7804
proposed structure for altair (#2323)
* proposed structure for hf1

* refactor datatypes.nim into datatypes/{base, phase0, hf1}.nim

* hf1 is Altair

* some syncing with alpha 2

* adjust epoch processing to disambiguate access to RewardFlags

* relocate StateData to stay consistent with meaning phase 0 StateData

* passes v1.1.0 alpha 5 SSZ consensus object tests

* Altair block header test fixtures work

* fix slash_validator() so that Altair attester slashings, proposer slashings, and voluntary exit textures work

* deposit operation Altair test fixtures work

* slot sanity and all but a couple epoch transition tests switched to Altair

* attestation Altair test fixtures work

* Altair block sanity test fixtures work

* add working altair sync committee tests

* improve workarounds for sum-types-across-modules Nim bug; incorporate SignedBeaconBlock root reconstuction to SSZ byte reader
2021-05-28 15:25:58 +00:00
Jacek Sieka 646923c3dd
add attestation stats tool to ncli_db (#2539)
This also makes future efforts to provide metrics and logs for
attestation efficiency easier

* Export rewards from epoch transition
* Use less memory for reward calculation (bool -> set[enum], field
alignment)
* Reuse reward memory when replaying, avoiding spike
* Allow replaying any range in ncli_db benchmark
2021-05-07 13:36:21 +02:00
Jacek Sieka 427c0f307c
avoid extraneous hash root calculation (#2537)
When applying a block, we'll currently compute a state root for the
state after slot processing but before block processing - this is
unnecessary when a block is being applied because the intermediate state
root is never observed.
2021-05-05 08:54:21 +02:00
Jacek Sieka ce49da6c0a
Introduce unittest2 and junit reports (#2522)
* Introduce unittest2 and junit reports

* fix XML path

* don't combine multiple CI runs

* fixup

* public combined report also

Co-authored-by: Ștefan Talpalaru <stefantalpalaru@yahoo.com>
2021-04-28 18:41:02 +02:00
Zahary Karadjov 14b2d4324d openarray -> openArray 2020-11-03 23:23:10 +02:00
Jacek Sieka 7c0b4d28d2 speed up reward/penalty calculation
Calculating rewards/penalties is slow due to how we compute sets of
attestations validators then use the sets for inclusion checks, to see
who attested. The dominant function during validated block processing /
epoch processing is hash set building and lookup.

This PR inverts the flow by removing the sets and creating a single
large validator status list, then applying all relevant state
attestations, then updating rewards and penalties.

This provides a 10x speedup to epoch processing which in turn speeds up
both empty slot and block processing - for example, on startup, we
replay all non-finalized blocks to prime fork choice - the same when
validating attestations or replaying states on reorg.
2020-10-23 19:23:36 +03:00
tersec 3d5f24f14c
stop discarding future epochs; remove a StateCache() construction (#1610)
* stop discarding non-existent future epochs during epoch state transitions; remove a pointless StateCache() construction in advance_slots()

* update nbench to pass StateCache to process_slots()
2020-09-07 15:04:33 +00:00
Jacek Sieka 48cebc7157
spec cleanups (#1379)
* add helper for beacon committee length (used for quickly validating
attestations)
* refactor some attestation checks to do cheap checks early
* validate attestation epoch before computing active validator set
* clean up documentation / comments
* fill state cache on demand
2020-07-27 16:04:44 +00:00
tersec 90708a8287
Prefer converting int` to uint64 and switch foo.len.uint64 to .len64 (#1375)
* avoid converting from uint64 to int, and where most feasible, int type conversion at all

* .len.uint64 -> .len64

* fix 32-bit compilation

* try keeping state_sim loop variable/bounds as int for 32-bit Azure

* len64 -> lenu64
2020-07-26 20:55:48 +02:00
Jacek Sieka e0a18a3105
cache beacon committee size calculation (#1363)
* cache beacon committee size calculation

this fixes a bug in get_validator_churn_limit as well

* fix

* make committee counts consistently uint64

mixing feels like the worst of the two worlds
2020-07-23 19:01:07 +02:00
tersec 6b77f3dda5
update compute_subnet_for_attestation() to use https://github.com/ethereum/eth2.0-specs/pull/1876 signature, which isn't in v0.12.1, which works with lookahead (#1346) 2020-07-22 08:04:21 +00:00
tersec 26e893ffc2
restore EpochRef and flush statecaches on epoch transitions (#1312)
* restore EpochRef and flush statecaches on epoch transitions

* more targeted cache invalidation

* remove get_empty_per_epoch_cache(); implement simpler but still faster get_beacon_proposer_index()/compute_proposer_index() approach; add some abstraction layer for accessing the shuffled validator indices cache

* reduce integer type conversions

* remove most of rest of integer type conversion in compute_proposer_index()
2020-07-15 12:44:18 +02:00
Jacek Sieka f3e92762e3
add tests for unviable blocks (#1271)
* add tests for unviable blocks

also enable finalization tests in all test configs - they're plenty fast
now
also fix newClone for non-rvo cases. sigh.

* fixes
2020-07-01 19:00:14 +02:00
tersec 807b920c19
state_transition implements the spec fairly directly (#1220) 2020-06-23 13:54:24 +00:00
tersec b7bb62d48b
use shuffled validator cache in more places; cleanups (#1095) 2020-06-03 07:42:08 +02:00
Jacek Sieka 18eca263b1
avoid stack allocations in some tests (#1079) 2020-05-28 10:28:14 +02:00
Dustin Brody 6c941b1ff7 spec v0.11.3 update 2020-05-19 17:16:55 +00:00
Zahary Karadjov a99977b772 Progress towards #991; Enable the distinct ssz.List type 2020-05-19 15:53:03 +03:00
tersec 093d298b2b
Increase finalization and finalization checking robustness (#990)
* fix some warnings related to beacon_node splitting; reimplement finalization verification more robustly; improve attestation pool block selection logic

* re-add missing import

* whitelist allowed state transition flags and make rollback/restore naming more consistent

* restore usage of update flags passed into skipAndUpdateState(...) in addition to the potential verifyFinalization flag

* switch rest of rollback -> restore
2020-05-09 12:43:15 +00:00
tersec 3a56ddc5c4
begin removing plain BeaconState versions of state transition functions (#951)
* remove near-duplicate code paths: process_slot(), process_slots(), and state_transition() for BeaconState are now wrappers around the HashedBeaconState versions

* convert tests/test_state_transition.nim to use HashedBeaconState

* convert mocking infrastructure and spec_block/epoch_processing tests to use HashedBeaconBlock, and remove thus unused process_slot*(state: var BeaconState)
2020-04-30 08:44:19 +02:00
Zahary Karadjov 4e9fa51ae9 Introduce BeaconNodeRef and use it in all the right places 2020-04-26 13:04:53 +03:00
Zahary Karadjov fdcbfdff05 Pass the test suite with a BeaconState ref type 2020-04-26 13:04:53 +03:00
Dustin Brody d559f4ee89 create distinct CommitteeIndex type 2020-04-15 09:59:49 +00:00
Zed 6ba7b4b117 Generate markdown test reports 2020-03-13 14:38:59 +00:00
Jacek Sieka 746659bdc6
bump stew, fix warnings (#655) 2019-12-20 14:25:33 +01:00
Dustin Brody 570de0839d measure/find slow tests (#624) 2019-12-05 11:27:00 +01:00
Dustin Brody 1494bcc262 replace links to Apache and MIT licenses with HTTPS versions (#592) 2019-11-25 16:30:02 +01:00
Dustin Brody df1d00f13a remove UnusedImport spam (#549)
* remove UnusedImport spam

* re-add random import to beacon_node
2019-11-14 11:47:55 +01:00
Dustin Brody d5ce142511 fix network sim finalization; remove get_attestation_data_slot(...); remove 2 more get_crosslink_committee(...) calls 2019-11-12 14:49:26 +00:00
Dustin Brody 949d735155 fix remaining test 0.9.0 test fixtures and switch back to non-transitiontest setup; remove get_committee_count(...); keep SHARD_COUNT consistent with MAX_COMMITTEES_PER_SLOT 2019-11-11 10:25:57 +00:00
Dustin Brody 68654848cb update get_attesting_indices(...) to 0.9.0; remove Crosslink from AttestationData to update AttestationData to 0.9.0; rm get_winning_crosslink_and_attesting_indices(...) and get_crosslink_deltas(...) 2019-11-08 13:23:08 +00:00
Dustin Brody 63e621c27d
initial 0.9.0 spec sync (#509)
* rename compute_epoch_of_slot(...) to compute_epoch_at_slot(...)

* remove some unnecessary imports; remove some crosslink-related code and tests; complete renaming of compute_epoch_of_slot(...) to compute_epoch_at_slot(...)

* rm more transfer-related code and tests; rm more unnecessary strutils imports

* rm remaining unused imports

* remove useless get_empty_per_epoch_cache(...)/compute_start_slot_of_epoch(...) calls

* rename compute_start_slot_of_epoch(...) to compute_start_slot_at_epoch(...)

* rename ACTIVATION_EXIT_DELAY to MAX_SEED_LOOKAHEAD

* update domain types to 0.9.0

* mark AttesterSlashing, IndexedAttestation, AttestationDataAndCustodyBit, DepositData, BeaconBlockHeader, Fork, integer_squareroot(...), and process_voluntary_exit(...) as 0.9.0

* mark increase_balance(...), decrease_balance(...), get_block_root(...), CheckPoint, Deposit, PendingAttestation, HistoricalBatch, is_active_validator(...), and is_slashable_attestation_data(...) as 0.9.0

* mark compute_activation_exit_epoch(...), bls_verify(...), Validator, get_active_validator_indices(...), get_current_epoch(...), get_total_active_balance(...), and get_previous_epoch(...) as 0.9.0

* mark get_block_root_at_slot(...), ProposerSlashing, get_domain(...), VoluntaryExit, mainnet preset Gwei values, minimal preset max operations, process_block_header(...), and is_slashable_validator(...) as 0.9.0

* mark makeWithdrawalCredentials(...), get_validator_churn_limit(...), get_total_balance(...), is_valid_indexed_attestation(...), bls_aggregate_pubkeys(...), initial genesis value/constants, Attestation, get_randao_mix(...), mainnet preset max operations per block constants, minimal preset Gwei values and time parameters, process_eth1_data(...), get_shuffled_seq(...), compute_committee(...), and process_slots(...) as 0.9.0; partially update get_indexed_attestation(...) to 0.9.0 by removing crosslink refs and associated tests

* mark initiate_validator_exit(...), process_registry_updates(...), BeaconBlock, Eth1Data, compute_domain(...), process_randao(...), process_attester_slashing(...), get_base_reward(...), and process_slot(...) as 0.9.0
2019-10-30 19:41:19 +00:00
Jacek Sieka 2a28a95cfc update test as well 2019-10-02 12:27:51 +03:00
Mamy André-Ratsimbazafy 763e296731
Fix and activate the justification and finalization unit tests. 2019-09-24 16:56:12 +02:00