Commit Graph

1003 Commits

Author SHA1 Message Date
tersec 1a6a56bdb1
use BeaconTime instead of Slot in fork choice (#3138)
* use v1.1.6 test vectors; use BeaconTime instead of Slot in fork choice

* tick through every slot at least once

* use div INTERVALS_PER_SLOT and use precomputed constants of them

* use correct (even if numerically equal) constant
2021-12-21 18:56:08 +00:00
tersec 0d4e49f946
Merge fork gossip support (#3213)
* Merge fork gossip support

* index directly by BeaconStateFork and remove debugging log statement
2021-12-21 15:24:23 +01:00
Jacek Sieka 1021e3324e
Revert writing backfill root to database (#3215)
Introduced in #3171, it turns out we can just follow the block headers
to achieve the same effect

* leaves the constant in the code so as to avoid confusion when reading
database that had the constant written (such as the fleet nodes and
other unstable users)
2021-12-21 11:40:14 +01:00
Jacek Sieka c270ec21e4
Validator monitoring (#2925)
Validator monitoring based on and mostly compatible with the
implementation in Lighthouse - tracks additional logs and metrics for
specified validators so as to stay on top on performance.

The implementation works more or less the following way:
* Validator pubkeys are singled out for monitoring - these can be
running on the node or not
* For every action that the validator takes, we record steps in the
process such as messages being seen on the network or published in the
API
* When the dust settles at the end of an epoch, we report the
information from one epoch before that, which coincides with the
balances being updated - this is a tradeoff between being correct
(waiting for finalization) and providing relevant information in a
timely manner)
2021-12-20 20:20:31 +01:00
tersec d7799ecdcc
v1.1.6 spec updates (#3206) 2021-12-17 06:56:33 +00:00
Jacek Sieka 118840d241
SyncManager cleanups for backfill support (#3189)
* SyncManager cleanups for backfill support

Cleanups, fixes and simplifications, in anticipation of backfill support
for the `SyncManager`:

* reformat sync progress indicator to show time left and % done more
prominently:
  * old: `sync="sPssPsssss:2:2.4229:00h57m (2706898)"`
  * new: `sync="14d12h31m (0.52%) 1.1378slots/s (wQQQQQDDQQ:1287520)"`
* reset average speed when going out of sync
* pass all block errors to sync manager, including duplicate/unviable
* penalize peers for reporting a head block that is outside of our
expected wall clock time (they're likely on a different network or
trying to disrupt sync)
* remove `SyncFailureKind` (unused)
* remove `inRange` (unused)
* add `Q` for sync queue requests that are in the `SyncQueue` but not
yet in the `BlockProcessor` queue
* update last slot in `SyncQueue` after getting peer status
* fix race condition between `wakeupWaiters` and `resetWait`, where
workers would not be correctly reset if block verification returned a
completed future without event loop
* log syncmanager direction

* Fix ordering issue.
Some of the requests size of which are not equal to `chunkSize` could be processed in wrong order which could lead to sync process freezes.

Co-authored-by: cheatfate <eugene.kabanov@status.im>
2021-12-16 15:57:16 +01:00
tersec 36ade1c1c6
v1.1.6 spec updates (minor, mostly URLs) (#3197) 2021-12-14 21:02:29 +00:00
tersec 4498d96a9a
don't build tests_blockchain_dag or tests_keystore on i386 (#3190) 2021-12-14 06:06:05 +00:00
tersec f09686e835
update some spec URLs to v1.1.6 (#3188) 2021-12-13 15:45:48 +00:00
Jacek Sieka 03005f48e1
Backfill support for ChainDAG (#3171)
In the ChainDAG, 3 block pointers are kept: genesis, tail and head. This
PR adds one more block pointer: the backfill block which represents the
block that has been backfilled so far.

When doing a checkpoint sync, a random block is given as starting point
- this is the tail block, and we require that the tail block has a
corresponding state.

When backfilling, we end up with blocks without corresponding states,
hence we cannot use `tail` as a backfill pointer - there is no state.

Nonetheless, we need to keep track of where we are in the backfill
process between restarts, such that we can answer GetBeaconBlocksByRange
requests.

This PR adds the basic support for backfill handling - it needs to be
integrated with backfill sync, and the REST API needs to be adjusted to
take advantage of the new backfilled blocks when responding to certain
requests.

Future work will also enable moving the tail in either direction:
* pruning means moving the tail forward in time and removing states
* backwards means recreating past states from genesis, such that
intermediate states are recreated step by step all the way to the tail -
at that point, tail, genesis and backfill will match up.
* backfilling is done when backfill != genesis - later, this will be the
WSS checkpoint instead
2021-12-13 14:36:06 +01:00
Jacek Sieka dfbd50b4d6
avoid SyncCommitteMsgPool copy (#3185)
introduced by batch verification, when verifiers were made async
2021-12-11 16:39:24 +01:00
Etan Kissling 984dc18dc6
import `is_valid_merkle_branch` test cases from `nim-eth` (#3182)
As of https://github.com/status-im/nim-eth/pull/379 `nim-eth` defines a
couple static test cases for merkle proof verification.
Since the EF has defined a `is_valid_merkle_branch` function in the spec
we are no longer using the custom implementation from `nim-eth`, but the
tests were never ported to target the new implementation. This patch now
follows up on that and integrates those tests from `nim-eth`.
2021-12-10 16:56:26 +01:00
Jacek Sieka 9f27f0d97c
BlockId reform (#3176)
* BlockId reform

Introduce `BlockId` that helps track a root/slot pair - this prepares
the codebase for backfilling and handling out-of-dag blocks

* move block dag code to separate module
* fix finalised state root in REST event stream
* fix finalised head computation on head update, when starting from
checkpoint
* clean up chaindag init
* revert `epochAncestor` change in introduced in #3144 that would return
an epoch ancestor from the canoncial history instead of the given
history, causing `EpochRef` keys to point to the wrong block
2021-12-09 19:06:21 +02:00
Etan Kissling 5cc6db5e20
remove disabled incorrect attestation test (#3175)
In #780 a test was disabled that verified that an attestation with
empty `aggregation_bits` completes successfully. The test was never
re-introduced, and as of the current consensus spec v1.1.6, such
attestations are not considered valid, as they fail the check in
`is_valid_indexed_attestation`. This patch fully removes that outdated
test, and moves it to the list of pending invalid attestation tests.
2021-12-09 14:03:22 +01:00
Jacek Sieka 069bccd51b
batch-verify sync messages for a small perf boost (#3151)
* batch-verify sync messages for a small perf boost

Generally reuses the same structure as attestation and aggregate
verification

* normalize `signatures` and `signature_batch` to use the same pattern
of verification
* normalize parameter names, order etc for signature stuff in general
* avoid calling `blsSign` directly - instead, go through `signatures`
consistently
2021-12-09 14:56:54 +02:00
tersec d93a279565
engine API alpha.5 field renaming (#3174) 2021-12-09 11:18:38 +00:00
Eugene Kabanov b05734f610
Backward sync support for SyncManager. (#3131)
* Unbundle SyncQueue from sync_manager.nim.
Unbundle Peer scores constants to peer_scores.nim.
Add Forward/Backward enum.

* Further improvements and tests.

* Adopt getRewindPoint() and fix MissingParent handler.

* Remove unused procedures.
Refactor `result` usage.
Fix resetWait().

* Add all the tests and fix the issue with rewind point.

* Fix get() issue.

* Fix flaky tests.

* test fixes

Co-authored-by: Jacek Sieka <jacek@status.im>
2021-12-08 22:15:29 +01:00
Jacek Sieka 89d6a1b403
Introduce slot->BlockRef mapping for finalized chain (#3144)
* Introduce slot->BlockRef mapping for finalized chain

The finalized chain is linear, thus we can use a seq to lookup blocks by
slot number.

Here, we introduce such a seq, even though in the future, it should
likely be backed by a database structure instead, or, more likely, a
flat era file with a flat lookup index.

This dramatically speeds up requests by slot, such as those coming from
the REST interface or GetBlocksByRange, as these are currently served by
a linear iteration from head.

* fix REST block requests to not return blocks from an earlier slot when
the given slot is empty
* fix StateId interpretation such that it doesn't treat state roots as
block roots
* don't load full block from database just to return its root
2021-12-06 20:52:35 +02:00
Jacek Sieka 1a8b7469e3
move quarantine outside of chaindag (#3124)
* move quarantine outside of chaindag

The quarantine has been part of the ChainDAG for the longest time, but
this design has a few issues:

* the function in which blocks are verified and added to the dag becomes
reentrant and therefore difficult to reason about - we're currently
using a stateful flag to work around it
* quarantined blocks bypass the processing queue leading to a processing
stampede
* the quarantine flow is unsuitable for orphaned attestations - these
should also should be quarantined eventually

Instead of processing the quarantine inside ChainDAG, this PR moves
re-queueing to `block_processor` which already is responsible for
dealing with follow-up work when a block is added to the dag

This sets the stage for keeping attestations in the quarantine as well.

Also:

* make `BlockError` `{.pure.}`
* avoid use of `ValidationResult` in block clearance (that's for gossip)
2021-12-06 10:49:01 +01:00
tersec a8c801eddd
fix Altair fork tests in minimal preset (#3163) 2021-12-06 05:56:46 +00:00
tersec e6921f808f
cleanups, partly from kintsugi branch (#3161)
* cleanups, partly from kintsugi branch

* re-export shortLog(EthBlock) and preserve exception messages in batchVerify and processBatch
2021-12-05 17:32:41 +00:00
tersec 4378f3f096
almost all remaining ethereum/{eth2.0-specs -> consensus-specs} (#3158) 2021-12-03 20:01:13 +00:00
tersec cc51f3fd12
v1.1.{5 -> 6} phase 0 and altair spec URL updates (#3157) 2021-12-03 17:40:23 +00:00
Zahary Karadjov 6fddff524c Switch back to WebSocket URLs in the Eth1Monitor
The HTTP support is not stable enough yet.
2021-12-03 17:04:29 +02:00
Etan Kissling 5e9625c1be extend `makeTestBlocks` for sync aggregates
This extends the `makeTestBlocks` function used in tests with a new
parameter `syncCommitteeRatio` to control whether the produced blocks
should be signed by the validators assigned to the sync committee.
A similar parameter already exists to configure whether attestations
for the test blocks should be produced.
2021-12-01 20:10:33 +02:00
tersec 61fb458f89
use v1.1.6 test vectors (#3146) 2021-12-01 12:55:42 +00:00
Jacek Sieka aa1dea03cd
speed up gossip and sync block validation (#3143)
* avoid recomputing hash for block signature check
* check block slot match before hitting the database
2021-12-01 10:52:40 +01:00
cheatfate b3ee5d67bd Remove nimbus_signing_process. 2021-11-30 16:48:36 +02:00
Eugene Kabanov e62c7c7c37
Remote signing client/server. (#3077) 2021-11-30 03:20:21 +02:00
Mamy Ratsimbazafy 97da6e1365
Fork choice EF consensus tests (#3041)
* add EF fork choice tests to CI

* checkpoints

* compilation fixes and add test to preset dependent suite

* support longpaths on Windows CI

* skip minimal tests (long paths issue + impl detals tested)

* fix stackoverflow on some platforms

* rebase on top of https://github.com/status-im/nimbus-eth2/pull/3054

* fix stack usage
2021-11-25 19:41:39 +01:00
Jacek Sieka a223d62b07
Cleanups (#3123)
Renames and cleanups split out from the validator monitoring branch, so
as to reduce conflict area vs other PR:s

* add constants for expected message timing
* name validators after the messages they validate, mostly, to make
grepping easier
* unify field naming of EpochInfo across forks to make cross-fork code
easier
2021-11-25 13:20:36 +01:00
Jacek Sieka 9c2f43ed0e
Speed up altair block processing 2x (#3115)
* Speed up altair block processing >2x

Like #3089, this PR drastially speeds up historical REST queries and
other long state replays.

* cache sync committee validator indices
* use ~80mb less memory for validator pubkey mappings
* batch-verify sync aggregate signature (fixes #2985)
* document sync committee hack with head block vs sync message block
* add batch signature verification failure tests

Before:

```
../env.sh nim c -d:release -r ncli_db --db:mainnet_0/db bench --start-slot:-1000
All time are ms
     Average,       StdDev,          Min,          Max,      Samples,         Test
Validation is turned off meaning that no BLS operations are performed
    5830.675,        0.000,     5830.675,     5830.675,            1, Initialize DB
       0.481,        1.878,        0.215,       59.167,          981, Load block from database
    8422.566,        0.000,     8422.566,     8422.566,            1, Load state from database
       6.996,        1.678,        0.042,       14.385,          969, Advance slot, non-epoch
      93.217,        8.318,       84.192,      122.209,           32, Advance slot, epoch
      20.513,       23.665,       11.510,      201.561,          981, Apply block, no slot processing
       0.000,        0.000,        0.000,        0.000,            0, Database load
       0.000,        0.000,        0.000,        0.000,            0, Database store
```

After:

```
    7081.422,        0.000,     7081.422,     7081.422,            1, Initialize DB
       0.553,        2.122,        0.175,       66.692,          981, Load block from database
    5439.446,        0.000,     5439.446,     5439.446,            1, Load state from database
       6.829,        1.575,        0.043,       12.156,          969, Advance slot, non-epoch
      94.716,        2.749,       88.395,      100.026,           32, Advance slot, epoch
      11.636,       23.766,        4.889,      205.250,          981, Apply block, no slot processing
       0.000,        0.000,        0.000,        0.000,            0, Database load
       0.000,        0.000,        0.000,        0.000,            0, Database store
```

* add comment
2021-11-24 13:43:50 +01:00
Zahary Karadjov 88c623e250 Add support for HTTPS Web3 providers 2021-11-23 15:56:18 +02:00
Jacek Sieka f19a497eec
ncli_db: add putState, putBlock (#3096)
* ncli_db: add putState, putBlock

These tools allow modifying an existing nimbus database for the purpose
of recovery or reorg, moving the head, tail and genesis to arbitrary
points.

* remove potentially expensive `putState` in `BeaconStateDB`
* introduce `latest_block_root` which computes the root of the latest
applied block from the `latest_block_header` field (instead of passing
it in separately)
* avoid some unnecessary BeaconState copies during init
* discover https://github.com/nim-lang/Nim/issues/19094
* prefer `HashedBeaconState` in a few places to avoid recomputing state
root
* fetch latest block root from state when creating blocks
* harden `get_beacon_proposer_index` against invalid slots and document
* move random spec function tests to `test_spec.nim`
* avoid unnecessary state root computation before block proposal
2021-11-18 13:02:43 +01:00
tersec 9e395011d9
update 22 spec URLs to v1.1.5 (#3111) 2021-11-18 08:08:00 +00:00
tersec 39f6a6534e
document how to run merge local testnet with Nethermind (#3110) 2021-11-17 20:45:39 +01:00
tersec c5fa7387f9
update kintsugi docs (#3104) 2021-11-15 17:01:27 +00:00
tersec fde73b5a70
bump nim-web3 to switch payload ID from uint64 to 8-byte buffer (#3103) 2021-11-15 14:55:03 +00:00
Zahary Karadjov 5e62df5ad5 /bin/bash -> /usr/bin/env bash (needed on NixOS) 2021-11-15 15:45:01 +02:00
Jacek Sieka 222674b203
better logging on invalid database (#3097)
* remove redundant test report
2021-11-13 17:27:28 +01:00
tersec 97ad5d49b0
kintsugi merge vector tests (#3072) 2021-11-10 13:41:02 +02:00
Jacek Sieka ec650c7fd7
Support starting from altair (#3054)
* Support starting from altair

* hide `finalized-checkpoint-` - they are incomplete and usage may cause
crashes
* remove genesis detection code (broken, obsolete)
* enable starting ChainDAG from altair checkpoints - this is a
prerequisite for checkpoint sync (TODO: backfill)
* tighten checkpoint state conditions
* show error when starting from checkpoint with existing database (not
supported)
* print rest-compatible JSON in ncli/state_sim
* altair/merge support in ncli
* more altair/merge support in ncli_db
* pre-load header to speed up loading
* fix forked block decoding
2021-11-10 13:39:08 +02:00
kdeme 03a70fbf36 Use nim-ssz-serialization module and rm local ssz code 2021-11-10 13:37:24 +02:00
tersec 5c48982280
a dozen spec URL updates to v1.1.5 (#3078) 2021-11-10 08:12:41 +00:00
tersec 941eb609ba
update statediffs to work with Altair (#3061)
* update statediffs to work with Altair
2021-11-09 16:17:57 +00:00
tersec 48eba59971
manually-verified v1.1.5 spec URL updates (#3068) 2021-11-09 08:54:59 +00:00
tersec 2e868dc2ba
mass/mechanical update of 1.1.4 phase0 and altair spec URLs to 1.1.5 (#3067) 2021-11-09 07:40:41 +00:00
tersec b76326f4b8
use v1.1.5 consensus spec test vectors (#3066) 2021-11-09 06:46:24 +00:00
tersec 5c296c08b7
17 spec URL updates to v1.1.4 (#3060) 2021-11-08 10:30:52 +00:00
tersec eb3ad25859
mass/mechanical update of 1.1.3 phase/altair spec URLs to 1.1.4 (#3058) 2021-11-08 06:18:10 +00:00
Dustin Brody 023a6a9387 use v1.1.4 test vectors 2021-11-07 21:42:11 +02:00
Zahary Karadjov 29e5700838 Bugfix: Avoid the aggregation of duplicate signatures when creating sync committee contributions 2021-11-07 21:41:10 +02:00
Jacek Sieka ea0a191723
Better REST/RPC error messages (#3046)
* Better REST/RPC error messages
* homogenise block logging (root first)
* homegenise message verification pipeline (verify in
`gossip_verification`, act in `eth2_processor`)
* use `subcommitteeIdx` consistently
* log each sent contribution
* fix block_sim
* fix block topic
* don't recalc root on gossip block validation
* move position loop into sync pool
2021-11-05 17:39:47 +02:00
Jacek Sieka a086cf01ac
altair fork handling cleanups (#3050)
* fix stack overflow crash in REST/debug/getStateV2
* introduce `ForkyXxx` for generic type matching of `Xxx` across
branches (SomeHashedBeaconState -> ForkyHashedBeaconState et al) -
`Some` is already used for other types of type classes
* consolidate function naming in BeaconChainDB, use some generics
* import `forks.nim` from other spec modules and move `Forked*` helpers
around to resolve circular imports
* remove `ForkedBeaconState`, use `ForkedHashedBeaconState` throughout
(less data shuffling between the types)
* fix several cases of states being stored on stack in tests, causing
random failures on some platforms
* remove reading json support from ncli - this should be ported to the
rest json reading instead (doesn't currently work because stack sizes)
2021-11-05 08:34:34 +01:00
Etan Kissling 51aef34be7
introduce strong typing for sync committee periods (#3040)
The sync committee period used to be a plain `uint64`. With the light
client sync relying more heavily on them, it makes sense to introduce
a proper type for them, similar to how they are already used for `Slot`
and `Epoch`. This introduces such a `SyncCommitteePeriod` type.
Furthermore, some usage code dealing with those periods is cleaned up.
2021-11-02 21:32:34 +01:00
Jacek Sieka 233d756518
Logging and startup improvements (#3038)
* Logging and startup improvements

Color support for released binaries!

* startup scripts no longer log to file by default - this only affects
source builds - released binaries don't support file logging
* add --log-stdout option to control logging to stdout (colors, json)
* detect tty:s vs redirected logs and log accordingly
* add option to disable log colors at runtime
* simplify several "common" logs, showing the most important information
earlier and more clearly
* remove line numbers / file information / tid - these take up space and
are of little use to end users
  * still enabled in debug builds and tools
* remove `testnet_servers_image` compile-time option
* server images, released binaries and compile-from-source now offer
the same behaviour and features
* fixes https://github.com/status-im/nimbus-eth2/issues/2326
* fixes https://github.com/status-im/nimbus-eth2/issues/1794
* remove instanteneous block speed from sync message, keeping only
average

before:

```
INF 2021-10-28 16:45:59.000+02:00 Slot start                                 topics="beacnde" tid=386429 file=nimbus_beacon_node.nim:884 lastSlot=2384027 wallSlot=2384028 delay=461us84ns peers=0 head=75a10ee5:3348 headEpoch=104 finalized=cd6804ba:3264 finalizedEpoch=102 sync="wwwwwwwwww:0:0.0000:0.0000:00h00m (3348)"
INF 2021-10-28 16:45:59.046+02:00 Slot end                                   topics="beacnde" tid=386429 file=nimbus_beacon_node.nim:821 slot=2384028 nextSlot=2384029 head=75a10ee5:3348 headEpoch=104 finalizedHead=cd6804ba:3264 finalizedEpoch=102 nextAttestationSlot=-1 nextProposalSlot=-1 nextActionWait=n/a
```

after:

```
INF 2021-10-28 22:43:23.033+02:00 Slot start                                 topics="beacnde" slot=2385815 epoch=74556 sync="DDPDDPUDDD:10:5.2258:01h19m (2361088)" peers=37 head=eacd2dae:2361096 finalized=73782:a4751487 delay=33ms687us715ns
INF 2021-10-28 22:43:23.291+02:00 Slot end                                   topics="beacnde" slot=2385815 nextActionWait=n/a nextAttestationSlot=-1 nextProposalSlot=-1 head=eacd2dae:2361096
```

* fix comment

* documentation updates

* mention `--log-file` may be deprecated in the future
* update various docs
2021-11-02 18:06:36 +01:00
tersec 36e37bda40
v1.1.3 spec refs URLs (#3036) 2021-10-27 18:40:17 +00:00
tersec 968a5b3bec
some spec refs to v1.1.3 (#3032) 2021-10-26 18:42:48 +00:00
tersec 8307e9c601
mechanical non-merge v1.1.2 to v1.1.3 spec URL updates (#3030) 2021-10-26 16:44:23 +00:00
Jacek Sieka 421bf936ff
odds and ends (#3015)
* `allSyncCommittees` => `allSyncSubcommittees`
* simplify `_snappy` topic generation (avoid pointless string copies)
* simplify gossip id generator (avoid pointless string copies)
* avoid redundant syncnet ENR updates
* simplify topic validation (allow only validated topics)
2021-10-21 15:09:19 +02:00
Etan Kissling 6720e6e2a6 fix running local validator client simulation
Updated the local network simulation scripts to use the REST interface
for connecting the validator client to the beacon node. Otherwise, the
current scripts produce errors.
2021-10-21 00:47:33 +03:00
Jacek Sieka 9cf32c3748 clean up sync subcommittee handling
* `SyncCommitteeIndex` -> `SyncSubcommitteeIndex`
* `syncCommitteePeriod` -> `sync_committee_period` (spec spelling)
* tighten period comparisons
* fix assert when validating committee message with non-altair state in
REST api
2021-10-20 22:59:13 +03:00
tersec c0a2f1c98e
refactor executionPayload tests; reduce HashSet creation (#3003) 2021-10-20 13:36:38 +02:00
tersec a6ddb80779
refactor EF operations fixtures (#2998)
* refactor EF operations fixtures

* for phase0/altair, check completeness of operations directories
2021-10-19 16:38:41 +02:00
Jacek Sieka df3fc9525f
import cleanup (#2997)
* import cleanup

...and remove some unused types

* add random imports

* more imports
2021-10-19 16:09:26 +02:00
Jacek Sieka c40cc6cec1 clean up fork enum and field names
* single naming strategy
* simplify some fork code
* simplify forked block production
2021-10-19 11:06:38 +03:00
tersec ecf2dbbcf3
add Altair and Merge rewards tests (#2995) 2021-10-18 16:54:17 +02:00
Jacek Sieka 4f7a8cf79d
register vc duties with subnet tracker (#2949)
* register vc duties with subnet tracker
* fix activation logging during startup
* cache slot signature to avoid duplicate signature work
* schedule aggregation duties one slot at a time to avoid CPU spike at
each epoch
* lower aggregation subnet pre-subscription time to 4 slots (lowers
bandwidth and CPU usage)
* update stability subnets in ENR on startup
* log gossip state
* perform gossip subscriptions just before the next slot starts
* document stuff
* add random include
* don't overwrite subscription state when not subscribed
* log target gossip state
* updating gossip status once is enough
* add test
* remove syncQueueLen - this one is not updated at the end of the sync
and may cause gossip to disconnect itself completely - use a simple head
distance instead
* fix gossip disconnection - if in hysteresis, node.gossipState will be
set to disabled even though we don't disable topic subscriptions
* fix extra duty registration call
2021-10-18 11:11:44 +02:00
tersec b66e7c8573
v1.1.3 tests; re-enable minimal-preset altair transition tests (#2993) 2021-10-16 07:17:07 +00:00
Etan Kissling 05908859b7
support merkle multiproof verification (#2980)
This introduces spec-compatible support to verify merkle multiproofs for
use, e.g., in light client syncing.
2021-10-14 10:28:22 +02:00
tersec 6cc8757930
update 18 spec URLs to v1.1.2; switch from eth2.0-specs to consensus-specs (#2990) 2021-10-14 06:30:21 +00:00
tersec 10981639f1
update 27 spec URLs to v1.1.2 (#2989) 2021-10-13 16:49:06 +00:00
Jacek Sieka f90b2b8b1f
reward accounting for altair+ (#2981)
Similar to the existing `RewardInfo`, this PR adds the infrastructure
needed to export epoch processing information from altair+. Because
accounting is done somewhat differently, the PR uses a fork-specific
object to extrct the information in order to make the cost on the spec
side low.

* RewardInfo -> EpochInfo, ForkedEpochInfo
* use array for computing new sync committee
* avoid repeated total active balance computations in block processing
* simplify proposer index check
* simplify epoch transition tests
* pre-compute base increment and reuse in epoch processing, and a few
other small optimizations

This PR introduces the type and does the heavy lifting in terms of
refactoring - the tools that use the accounting will need separate PR:s
(as well as refinements to the exportred information)
2021-10-13 16:24:36 +02:00
tersec 2eb9a608a4
add payloadId; add merge vector test script; remove consensusValidated (#2982) 2021-10-13 16:08:50 +02:00
Jacek Sieka 0c4bfb1911
test fixture cleanups (#2976)
* "official" -> "scenarios", like the submodule
* fewer test binaries - various compile hacks have been improved over
time, test suite should follow
* remove obsolete bls tests - there are better test vectors in
nim-blscurve
* remove obsolete mentions of `ssz_testing`
* remove obsolete comments about proc vs globals, unittest2 already uses
proc's
2021-10-12 13:36:52 +02:00
tersec 2ad1b7366a
update 62 spec URLs to v1.1.2 (#2979) 2021-10-12 10:17:37 +00:00
tersec 0ae736f397
update 67 spec URLs to v1.1.2 (#2977) 2021-10-12 08:09:59 +00:00
tersec 3c14e5728d
use v1.1.2 test vectors (#2968) 2021-10-08 11:36:35 +00:00
Jacek Sieka fabec894dd
harden sync sub committee selection (#2965)
* harden sync sub committee selection

also turn it into an iterator

* fix test and warning
2021-10-07 13:19:47 +00:00
Etan Kissling dfb87f1d5b
allow `mock_blocks` to create Merge blocks (#2963)
This extends `mock_blocks` support to also allow producing Merge blocks.
The only step missing was initialization of the `ExecutionPayload` to an
empty value (as created by `build_empty_execution_payload`).
2021-10-07 11:28:09 +02:00
Etan Kissling 9ee134324b
allow `withXxx` to access fork-specific fields (#2943)
So far, `withState` and `withBlck` templates could only be used to have
convenience access to fork-agnostic BeaconState and BeaconBlock fields.
This patch:
- injects an additional `stateFork` constant that allows to use
  `when` expressions to also access Altair and Merge-specific fields.
- introduces a `withStateAndBlck` template to support operating on both
  a `BeaconState` and `BeaconBlock` at a time.
- makes sync committee related functions Merge aware.
- changes a couple if-else trees for forks into case statements so that
  forgotten future forks are promoted to compile-time errors.
2021-10-06 20:05:06 +03:00
Etan Kissling 55c2553193
enable uint128 / uint256 SSZ serialization tests (#2960)
So far, SSZ tests for `uint128` and `uint256` integers were disabled
due to "compile-time issues". The code has been adjusted to support
those additional test vectors, and the corresponding tests have been
enabled.
2021-10-06 06:14:52 +00:00
tersec 5cc2acb44e
use v1.1.1 test vectors (#2954) 2021-10-05 02:26:56 +00:00
Eugene Kabanov 65257b82f8
Validator key management API (#2755)
Implements https://github.com/ethereum/beacon-APIs/pull/151
2021-10-04 22:08:31 +03:00
Etan Kissling f10a2b18bb port EF light client tests
EF has a number of unit tests related to light client functionality, see
https://github.com/ethereum/consensus-specs/blob/v1.1.0/tests/core/pyspec/eth2spec/test/altair/unittests/test_sync_protocol.py
The tests have been ported over and integrated into our test suite.
2021-10-04 15:49:34 +03:00
Etan Kissling f8e9b1ff9d
remove privkey from mock withdrawal credentials (#2936)
In tests, the private key was put into the validator deposit's withdraw
credentials so that it can be recovered later. This leads to problems
when creating the validators through other means that do not put the key
there. In general, mock private keys only depend on the validator index,
though, and because it is clear what the index of a validator is, it is
not actually needed to put the key into the credentials.
2021-10-01 13:35:16 +02:00
Etan Kissling b217150f1d
use forked `getAttestationsForBlock` everywhere (#2937)
There are a number of locations in the code that get attestations on a
forked beacon state. For attestation pools test, a convenience wrapper
was available to reduce clutter. This patch integrates that wrapper into
the core component so that it can also take advantage of the wrapper.
2021-10-01 01:29:32 +00:00
Etan Kissling 6fc1511866
avoid `-Present` in copyrights (#2935)
Replaced non-conventional `-Present` form in copyright headers with the
more appropriate `-2021` version that is already used by the majority of
the code base. Copyright header should indicate the years during which
the specific file was modified.
2021-09-30 16:28:02 +00:00
Etan Kissling 9ee6c3e7ef
always use static keys in tests (#2928)
There are multiple ways how states can be processed during tests. One is
through tests/mocking, the other through tests/xxxutil. Both methods can
create genesis states with a set of deposits from mock validators. The
private keys for those validators are created differently, though. While
tests/mocking generates random keys, testblockutil generates fixed keys.
This is a problem when mixing both tests/mocking and testblockutil in a
single test case, as they assume that the deposits were generated using
the same method. This patch aligns the key generation across tests.
2021-09-30 17:14:03 +02:00
Etan Kissling 2e9fa87f8b
use `SyncAggregate.init()` everywhere (#2932)
The initialization of a `SyncAggregate` to its default value is not very
intuitive. There is an `init` function in `sync_committee_msg_pool` that
provides a convenience wrapper. This patch exports that initializer so
that the rest of the code base can also take advantage of it.
2021-09-30 13:56:07 +00:00
tersec 6b3bf7eb7b
merge hardfork database support (#2911)
* merge hardfork database support

* working block_sim

* recreate state transition changes
2021-09-30 01:07:24 +00:00
Etan Kissling 711c08f804
add function to build merkle proofs (#2874)
Adds a function that constructs a Merkle proof for a generalized index.
This will be used during light client sync to update light clients with
a new state (see NEXT_SYNC_COMMITTEE_INDEX / FINALIZED_ROOT_INDEX).
2021-09-29 15:02:34 +02:00
Etan Kissling e243ba2c0b
revise `makeBeaconBlock` overloads (#2879)
The phase0 and altair overloads of `makeBeaconBlock` slightly differ in
their signatures which makes using them unnecessarily verbose.
- A placeholder `sync_aggregate` argument similar to `executionPayload`
  is added to the phase0 overload to match the altair signature.
- A wrapper operating on `ForkedHashedBeaconState` is introduced.
2021-09-29 12:10:44 +00:00
Tanguy 06df40454f
Improve discovery during fork (#2913) 2021-09-29 14:06:16 +03:00
Etan Kissling a0e7463732
fix `mockBlock` for current slot (#2915)
In #2867 a regression was introduced that broke `mockBlock` when used to
compute a block for the current state. This patch fixes the regression.
2021-09-28 16:27:44 +00:00
tersec ca4c6b4c5c
update 30 spec URLs to v1.1.0 (#2914) 2021-09-28 14:01:46 +00:00
tersec aec5cf2b1b
update 31 spec reference URLs to v1.1.0 (#2910) 2021-09-28 07:46:06 +00:00
Etan Kissling 01a9b275ec
handle duplicate pubkeys in sync committee (#2902)
When sync committee message handling was introduced in #2830, the edge
case of the same validator being selected multiple times as part of a
sync subcommittee was not covered. Not handling that edge case makes
sync contributions have a lower-than-expected participation rate as each
sync validator is only counted up through once per subcommittee.
This patch ensures that this edge case is properly covered.
2021-09-28 07:44:20 +00:00
tersec f2df006ee9
update to v1.1.0 (#2909) 2021-09-28 04:28:33 +00:00
tersec 2b2846b468
implement forked merge state/block support (#2890)
* implement forked state/block support

* merge support for containsOrphan; import cleanup; 80-column lines

* add merge block header operations and slot sanity fixture

* add epoch state transition tests; implement is_valid_gas_limit(), is_merge_block(), is_execution_enabled(), and compute_timestamp_at_slot()

* implement process_execution_payload() and add merge deposit operations tests

* add merge block sanity tests

* add merge case to syncCommitteeParticipants

* v1.1.0-beta.5 updates

* reduce getTestStates-based memory usage; don't try to REST-serialize ExecutionPayload transactions without underlying support

* add execution payload tests; switch var to let in tests/official/
2021-09-27 14:22:58 +00:00