Commit Graph

436 Commits

Author SHA1 Message Date
Dustin Brody db397cc5f3 lazily initialize validator public keys, which were a third or more of sync CPU usage 2020-08-04 23:00:33 +03:00
tersec a979568d74
switch mapIt/find to in-place linear search (#1436) 2020-08-04 17:15:27 +00:00
Jacek Sieka cf8cd8321b
Revert "Lazy crypto [alt design #1369] (#1371)" (#1435)
This reverts commit 023f7f4518.
2020-08-04 17:15:13 +00:00
tersec df80071bcf
update attestation and block validation to v0.12.2; clean up getAncestorAt()/get_ancestor() (#1417)
* update attestation validation to v0.12.2; clean up getAncestorAt()/get_ancestor()

* update beacon block validation to v0.12.2
2020-08-03 19:47:42 +00:00
tersec 99f2d8e06c
update 14 v0.12.1 spec refs to v0.12.2 (#1400) 2020-07-30 09:47:57 +00:00
Mamy Ratsimbazafy 023f7f4518
Lazy crypto [alt design #1369] (#1371)
* Lazy loading of crypto objects

* Try to fix incorrect field access by hiding fields but no luck. SSZ/Chronicles/macro bug?

* Fix incorrect blsValue access. was "aggregate" not "chronicles"

* Fix tests that rely on the internal BLSValue representation
2020-07-29 18:13:05 +00:00
Jacek Sieka 157ddd2ac4
Fork choice fixes 5 (#1381)
* limit attestations kept in attestation pool

With fork choice updated, the attestation pool only needs to keep track
of attestations that will eventually end up in blocks - we can thus
limit the horizon of attestations that we keep more aggressively.

To get here, we expose getEpochRef which gets metadata about a
particular epochref, and make sure to populate it when a block is added
- this ensures that state rewinds during block addition are minimized.

In addition, we'll use the target root/epoch when validating
attestations - this helps minimize the number of different states that
we need to rewind to, in general.

* remove CandidateChains.justifiedState

unused

* remove BlockPools.Head object

* avoid quadratic quarantine loop

* fix
2020-07-28 13:54:32 +00:00
tersec dad3dd5809
update 111 v0.12.1 beacon chain spec refs to v0.12.2 (#1380) 2020-07-27 18:49:46 +02:00
Jacek Sieka 48cebc7157
spec cleanups (#1379)
* add helper for beacon committee length (used for quickly validating
attestations)
* refactor some attestation checks to do cheap checks early
* validate attestation epoch before computing active validator set
* clean up documentation / comments
* fill state cache on demand
2020-07-27 16:04:44 +00:00
tersec 20a2525390
v0.12.2 beacon chain protocol update (#1378) 2020-07-27 12:59:57 +02:00
tersec 90708a8287
Prefer converting int` to uint64 and switch foo.len.uint64 to .len64 (#1375)
* avoid converting from uint64 to int, and where most feasible, int type conversion at all

* .len.uint64 -> .len64

* fix 32-bit compilation

* try keeping state_sim loop variable/bounds as int for 32-bit Azure

* len64 -> lenu64
2020-07-26 20:55:48 +02:00
Jacek Sieka e0a18a3105
cache beacon committee size calculation (#1363)
* cache beacon committee size calculation

this fixes a bug in get_validator_churn_limit as well

* fix

* make committee counts consistently uint64

mixing feels like the worst of the two worlds
2020-07-23 19:01:07 +02:00
Jacek Sieka fb2f742972
Fork choice fixes 2 (#1356)
* fork choice cleanup

* enable v2 pruning
* prefer `get_current_epoch`
* fix finalization check to use correct epoch

* small cleanups

* add `count_active_validators`
* remove misleading logs
* fix justified checkpoint slot calculation in rpc
2020-07-22 23:01:44 +02:00
tersec 4a9a7be271
faster syncing (#1348)
* maybe faster syncing

* 80-character lines

* remove instrumentation debugEchos; fix target attestation epoch in attestation pool validation

* use the epoch-granularity matching in attestation.addResolved(...)
2020-07-22 09:51:45 +02:00
tersec 83abbcb917
drop get_attesting_indices()/get_unslashed_attesting_indices() from 15% to 1% of workload at block_sim at 100k validators (#1351) 2020-07-21 18:35:43 +02:00
Jacek Sieka 8b01284b0e
cache block hash (#1329)
hash_tree_root was turning up when running beacon_node, turns out to be
repeated hash_tree_root invocations - this pr brings them back down to
normal.

this PR caches the root of a block in the SignedBeaconBlock object -
this has the potential downside that even invalid blocks will be hashed
(as part of deserialization) - later, one could imagine delaying this
until checks have passed

there's also some cleanup of the `cat=` logs which were applied randomly
and haphazardly, and to a large degree are duplicated by other
information in the log statements - in particular, topics fulfill the
same role
2020-07-16 15:16:51 +02:00
tersec 26e893ffc2
restore EpochRef and flush statecaches on epoch transitions (#1312)
* restore EpochRef and flush statecaches on epoch transitions

* more targeted cache invalidation

* remove get_empty_per_epoch_cache(); implement simpler but still faster get_beacon_proposer_index()/compute_proposer_index() approach; add some abstraction layer for accessing the shuffled validator indices cache

* reduce integer type conversions

* remove most of rest of integer type conversion in compute_proposer_index()
2020-07-15 12:44:18 +02:00
zah 0be77f9cbc
Restore compilation with custom presets (#1309)
This also assigns precise types to the constants in the minimal
and mainnet presets in order to reduce the chance of compilation
errors when custom presets are used (previously, only the custom
presets have precisely assigned types for the constants).
2020-07-13 16:44:58 +02:00
Zahary Karadjov 3ec6a02b12
Merge devel and resolve conflicts 2020-07-10 02:02:40 +03:00
tersec 61b0b5af17
update most remaining non-fork-choice spec refs, updating code where necessary (#1292)
* update most of the remaining non-fork-choice spec refs, updating code where necessary

* revert presumably harmless compute_signing_root() change, but this way, keep things really unchanged outside inspector
2020-07-09 11:43:27 +00:00
Mamy Ratsimbazafy 3cdae9f6be
Dual headed fork choice [Revolution] (#1238)
* Dual headed fork choice

* fix finalizedEpoch not moving

* reduce fork choice verbosity

* Add failing tests due to pruning

* Properly handle duplicate blocks in sync

* test_block_pool also add a test for duplicate blocks

* comments addressing review

* Fix fork choice v2, was missing integrating block proposed

* remove a spurious debug writeStackTrace

* update block_sim

* Use OrderedTable to ensure that we always load parents before children in fork choice

* Load the DAG data in fork choice at init if there is some (can sync witti)

* Cluster of quarantined blocks were not properly added to the fork choice

* Workaround async gcsafe warnings

* Update blockpoool tests

* Do the callback before clearing the quarantine

* Revert OrderedTable, implement topological sort of DAG, allow forkChoice to be initialized from arbitrary finalized heads

* Make it work with latest devel - Altona readyness

* Add a recovery mechanism when forkchoice desyncs with blockpool

* add the current problematic node to the stack

* Fix rebase indentation bug (but still producing invalid block)

* Fix cache at epoch boundaries and lateBlock addition
2020-07-09 11:29:32 +02:00
Dustin Brody 4140b3b9d9 update 29 spec refs to v0.12.1 2020-07-08 20:49:25 +00:00
Zahary Karadjov 318b225ccd
Merge devel and resolve the conflicts 2020-07-08 15:36:03 +03:00
Dustin Brody fc8502c54e halve memory usage from state caches 2020-07-08 10:21:41 +00:00
Zahary Karadjov c4af4e2f35
Working test suite with run-time presets 2020-07-08 02:02:14 +03:00
Zahary Karadjov 441ae9b58c
Introduce a RuntimePreset object for the overridable testnet preset values 2020-07-03 22:29:23 +03:00
tersec a92276d510
adopt Result[void, string] in place of some bool return signatures (#1275)
* adopt Result[void, string] in place of some bool return signatures

* string -> cstring to reduce memory allocations; ensure all err() strings are constants, with contextual information from higher-level callers

* logScope usage fixes

* homogenize err() reporting convention

* invalid signature in deposit isn't an error
2020-07-03 17:03:14 +00:00
Zahary Karadjov 31e31bb30c Switch to monitoring of Blocks; More accurate genesis detection and faster Eth1 syncing 2020-06-29 02:18:48 +03:00
zah 578ea2013a
implement spev v0.12.1 in initialize_beacon_state_from_eth1 (#1236) 2020-06-25 16:24:55 +02:00
Jacek Sieka 1301600341
Trusted blocks (#1227)
* cleanups

* fix ncli state root check flag
* add block dump to ncli_db
* limit ncli_db benchmark length
* tone down finalization logs

* introduce trusted blocks

We only store blocks whose signature we've verified in the database - as
such, there's no need to check it again, and most importantly, no need
to deserialize the signature when loading from database.

50x startup time improvement, 200x block load time improvement.

* fix rewinding when deposits have invalid signature
* speed up ancestor iteration by avoiding copy
* avoid deserializing signatures for trusted data
* load blocks lazily when rewinding (less memory used)

* chronicles workarounds

* document trustedbeaconblock
2020-06-25 12:23:10 +02:00
Mamy Ratsimbazafy 902093f57c
Revert "Dual headed fork choice [Reloaded] (#1223)" (#1234)
This reverts commit 6836d41ebd.
2020-06-25 11:36:03 +02:00
Mamy Ratsimbazafy 6836d41ebd
Dual headed fork choice [Reloaded] (#1223)
* Dual headed fork choice

* fix finalizedEpoch not moving

* reduce fork choice verbosity

* Add failing tests due to pruning

* Properly handle duplicate blocks in sync

* test_block_pool also add a test for duplicate blocks

* comments addressing review
2020-06-24 20:24:36 +02:00
Dustin Brody 62a7e7fede fix is_valid_indexed_attestation() to check for too-high attestation indices 2020-06-22 14:15:51 +00:00
Dustin Brody 4ecbc655ea switch 11 beaconstate.nim spec refs from v0.11.x to v0.12.1 2020-06-19 11:54:37 +00:00
tersec dc1a565b3f
support v0.12.1 attestation topics in beacon node/inspector subscribing (#1187)
* support v0.12.1 attestation topics in beacon node and inspector subscribing

* bump is_valid_merkle_branch() spec ref
2020-06-18 15:10:25 +02:00
tersec ee9f4a2e3f
remove skipMerkleValidation and skipBlockParentRootValidation (#1197) 2020-06-18 07:56:47 +02:00
Jacek Sieka 5c25d23ef1 eth2hash -> eth2digest
hash in nim is the insecure hash-map helper - might as well use `digest`
consistently to mark the difference
2020-06-16 13:30:21 +00:00
Jacek Sieka 89e4819ce9
collect signature production and verificaiton in one place (#1179)
* collect signature production and verificaiton in one place

Signatures are made over data and domain - here we collect all such
activities in one place.

Also:
* security: fix cast-before-range-check
* log block/attestation verification consistently
* run block verification based on `getProposer` in its own history
* clean up some unused stuff

* import

* missing raises
2020-06-16 07:45:04 +02:00
Dustin Brody 1d1e90adf4 Update 36 spec refs from 0.11.x to 0.12.1 2020-06-12 19:41:03 +00:00
Jacek Sieka 42832cefa8
Small fixes (#1165)
* random fixes

* create dump dir on startup
* don't crash on failure to write dump
* fix a few `uint64` instances being used when indexing arrays - this
should be a compile error but isn't due to compiler bugs
* fix standalone test_block_pool compilation
* add signed block processing in ncli

* reuse cache entry instead of allocating a new one

* allow for small clock disparities when validating blocks
2020-06-12 18:43:20 +02:00
Mamy Ratsimbazafy ce897fe83f
[Split fork choice PR] Derisk-ed attestation checks changes (#1154)
* Derisked attestation pool improvements

* tune down frequent logs

* VoteTracker logging
2020-06-10 08:58:12 +02:00
Jacek Sieka 56ffb696be
reorder ssz (#1099)
* reorder ssz

* split into hash_trees and ssz_serialization, roughly, for hashing and
IO
* move bitseqs into ssz (from stew)
* clean up imports

* docs, imports
2020-06-03 15:52:02 +02:00
tersec b7bb62d48b
use shuffled validator cache in more places; cleanups (#1095) 2020-06-03 07:42:08 +02:00
Zahary Karadjov a8003e7e38 More complete integration of the List type; Detect more invalid inputs; Enable more tests 2020-05-24 19:00:34 +03:00
Dustin Brody b9817867f5 audit & update spec references for some slashing and rewards and penalty functions; check and update all remaining v0.11.0 spec references to v0.11.3 [skip ci] 2020-05-20 14:57:08 +00:00
Dustin Brody 3fa2f70853 update all remaining beacon chain spec references from v0.11.1 and v0.11.2 to v0.11.3 2020-05-20 08:48:18 +00:00
Zahary Karadjov a99977b772 Progress towards #991; Enable the distinct ssz.List type 2020-05-19 15:53:03 +03:00
tersec 8432932c11
update to spec v0.11.2 to keep compatible with Schlesi (#1036)
* update to spec v0.11.2 to keep compatible with Schlesi

* update spec/state_transition_epoch spec references to v0.11.2

* bump other spec refs
2020-05-18 23:05:27 +00:00
Jacek Sieka fb2e0ddbec
sync fixes (#1005)
* sync fixes

* fix Status message finalized info
* work around sync starting before initial status exchange
* don't fail block on deposit signature check failure (fixes #989)
* print ForkDigest and Version nicely
* dump incoming blocks
* fix crash when libp2p peer connection is closed
* update chunk size to 16 to work around missing blocks when syncing

* bump libp2p

* bump libp2p

* better deposit skip message
2020-05-11 18:08:52 +00:00
tersec 093d298b2b
Increase finalization and finalization checking robustness (#990)
* fix some warnings related to beacon_node splitting; reimplement finalization verification more robustly; improve attestation pool block selection logic

* re-add missing import

* whitelist allowed state transition flags and make rollback/restore naming more consistent

* restore usage of update flags passed into skipAndUpdateState(...) in addition to the potential verifyFinalization flag

* switch rest of rollback -> restore
2020-05-09 12:43:15 +00:00
tersec c548b90a0e
default preset update; rm process_slots(BeaconState); some BLS skipping disabled (#963)
* re-enable BLS validation for block sanity tests

* update default preset to mainnet; remove process_slots(state: var BeaconState)

* bump spec ref to v0.11.1

* bump all v0.11.1 spec references except the actually changed is_valid_indexed_attestation(...) in spec/beaconstate to v0.11.2

* remove stack-risky/low-ROI doAssert
2020-05-04 21:07:18 +00:00
tersec cd939d4606
fix spec/beaconstate compile error in the presence of other PRs' SSZ tightening (#952) 2020-04-30 09:01:55 +00:00
tersec 3a56ddc5c4
begin removing plain BeaconState versions of state transition functions (#951)
* remove near-duplicate code paths: process_slot(), process_slots(), and state_transition() for BeaconState are now wrappers around the HashedBeaconState versions

* convert tests/test_state_transition.nim to use HashedBeaconState

* convert mocking infrastructure and spec_block/epoch_processing tests to use HashedBeaconBlock, and remove thus unused process_slot*(state: var BeaconState)
2020-04-30 08:44:19 +02:00
Zahary Karadjov 4e9fa51ae9 Introduce BeaconNodeRef and use it in all the right places 2020-04-26 13:04:53 +03:00
tersec 65353bab76
add another check for inconsistent aggregation and committee length (#927)
* add another check for inconsistent aggregation and committee length, since ncli_transition bypasses process_attestation(...)/check_attestation(...) and calls almost directly into process_epoch(...)

* bump validator functions to v0.11.1 spec references

* bump some spec references to v0.11.1

* poke
2020-04-24 09:16:40 +02:00
Jacek Sieka ed74770451
spec: regulate exceptions (#913)
* spec: regulate exceptions

* a few more simple raises
2020-04-22 07:53:02 +02:00
tersec 480c11f8aa
fix remaining block sanity test vectors (#902)
* fix remaining block sanity test vectors

* bump epoch processing spec refs to v0.11.1
2020-04-18 00:09:34 +02:00
Dustin Brody 164d171bd9
revert previous commit 2020-04-17 20:27:42 +02:00
Dustin Brody ef92529feb
fix remaining block sanity test vectors 2020-04-17 20:25:18 +02:00
Dustin Brody 3df9a22ff7 fix remaining attester slashing tests from EF phase0 spec tests by checking for uniqueness 2020-04-17 16:01:32 +00:00
Dustin Brody 05d1d1246a bump a couple more eth2 phase0 spec refs to v0.11.1 2020-04-17 16:37:52 +03:00
Dustin Brody d559f4ee89 create distinct CommitteeIndex type 2020-04-15 09:59:49 +00:00
Dustin Brody 2771deadfc re-add test_interop to all_tests and mark several v0.10.1 phase 0 spec references as v0.11.1 2020-04-07 13:16:55 +03:00
Dustin Brody e7de8aae20 fix/restore 0.11.1 BeaconState genesis interop: (a) use GENESIS_FORK_VERSION properly in BeaconState init; (b) use GENESIS_FORK_VERSION to be compatible with zcli/zrnt initial eth1 deposits; (c) let List[foo] work with 64-bit ints, so BeaconState.validators SSZ-serializes properly; and (d) update test_interop to use new spec/crypto API 2020-04-07 13:16:55 +03:00
Dustin Brody 60df05a420 mark several v0.10.1 spec references as v0.11.1-compatible 2020-04-04 13:43:04 +00:00
tersec ea59f2ecf2
comments-only changes: update a bunch of v0.10.1 spec references to v0.11.0 and explain rationale for extended validation design in code (#847) 2020-04-01 11:41:39 +00:00
tersec cd388bc9bb
extended validation (#812)
* initial extended validation setup

* flesh out all TODO items for attestation and beaconblock verification

* fix finalization and add chronicles debugging messages

* directly use blockPool.headState rather than pointlessly updating it and document this constraint

* fix logic relating to first-attestation checking; support validating blocks across multiple forks
2020-03-31 18:39:02 +00:00
tersec daabb1b5b2
complete all (non-semantic, comment-only) 0.11.0 -> 0.11.1 beacon chain spec updates, mark all beacon chain v0.11.0 spec references as v0.11.1, and remove now unnecessary/unused UpdateFlags vars from 4 test fixtures (#841) 2020-03-30 23:40:24 +00:00
tersec f5f939bd31
0.11.1 beacon chain spec update (#836)
* initial 0.11.1 spec commit; no test regressions and finalizes in eth2_network_simulation

* with BLS 0.10/0.11 available, stop skipping attester slashing, proposer slashing, and voluntary exist operations fixture tests

* switch param orders to group state.{fork, genesis_validators_root}; bump spec/datatypes spec version for network purposes

* mark attestation construction and broadcast and some minimal/mainnet constants as 0.11.1-compatible; remove phase 1 sharding constants from minimal which don't exist in that preset
2020-03-30 11:31:44 +00:00
Jacek Sieka 689bcf71c4 clean up block creation
* consistently use state at new block slot to produce block
* factor out signature funcs
* fix missing block application test
2020-03-22 11:15:07 +00:00
Dustin Brody cae1c0fd1a address beacon_node TODO regarding missing proposer index; mark initiate_validator_exit(...), is_eligible_for_activation_queue(...), Attestation, DepositMessage, BeaconBlockBody, PendingAttestation, Eth1Data, and integer_squareroot(...) as 0.11.0 2020-03-17 12:06:56 +02:00
Dustin Brody ebcde664e5 mark decrease_balance(...), get_validator_churn_limit(...), network spec version, Checkpoint, compute_epoch_at_slot(...), get_active_validator_indices(...), and get_committee_count_at_slot(...) as 0.11.0 2020-03-17 12:06:56 +02:00
Dustin Brody 6068dd960b initial 0.11.0 spec version update 2020-03-17 12:06:56 +02:00
Mamy André-Ratsimbazafy bfd4df912e Update to BLS v0.10.1 2020-03-13 12:20:41 +00:00
Jacek Sieka 9a3db7a81f
[WIP] Fake bls at runtime (#735)
* Initial implementation of runtime bls skipping.

Add libnfuzz skipBLSValidation handling, check that it propagates.

* Rename skipBLSValidation -> skipBlsValidation, start skipStateRootValidation
* Replace skipValidation flags with more granular flags.

Also added skipBlockParentRootValidation flag
Mainly replaced with skipBlsValidation but also StateRoot or
BlockParentRootValidation flags where appropriate.

* Adjust interop test to pass when skipping merkle validation.
* Stop skipping validation for mainchain_monitor.
* Remove comment.
* Also skipMerkleValidation for test_beacon_chain_db.
2020-03-05 13:52:10 +01:00
Dustin Brody a8567cafb5 mark out where/how AttestationData and Attestation construction are honest validator-compliant 2020-02-21 19:44:03 +00:00
Dustin Brody 1ffc2df23d add a couple new deposit tests; fix the false-positive BLS verifications while keeping all but two tests working, despite mismatched 0.9/0.10 BLS standards; better-factor the skipping of BLS validation and Merkle tree validation 2020-02-04 18:39:38 +00:00
Dustin Brody e4922cb177 implement compute_signing_root() and mark some functions as 0.10.1-compatible 2020-02-04 12:24:30 +00:00
Dustin Brody 45dd12cf3f update process_deposit() to actually check is_valid_merkle_branch() unless skipValidation specified 2020-01-30 09:31:08 +00:00
Dustin Brody 60b1775879 update check_attestation/process_attestation() to 0.10.1 and fill in missing check 2020-01-28 12:07:33 +02:00
Dustin Brody 3efbe36c30 additional 0.10.1 updates where already-implemented code is already done 2020-01-27 12:50:54 +00:00
Mamy Ratsimbazafy 1ba2cebd85 v0.10.1 (test vectors + unchanged parts) (#696)
* update to 0.10.1

* SSZ Generic and nbench uses the v0.10.1 fixtures

* Tests + spec links: v0.10.0 -> v0.10.1

* Add v0.10.1 TODO in get_latest_attesting_balance (forkchoice)

* SSZ Bytes are now ByteList

* Remove nim-result submodules that was leftover/added by mistake in the branch
2020-01-27 10:56:32 +00:00
Dustin Brody c301e5d42f mark 49 0.9.2/0.9.4 spec functions/data types as 0.10.0 2020-01-15 12:34:16 +00:00
Dustin Brody 46de726e3d mark all compatible 0.9.3 functions and data structures as 0.9.4 2019-12-20 17:24:42 +00:00
Mamy Ratsimbazafy 106352aff3
Nbench - Flexible benchmarking of Nimbus internals (#641)
* nbench PoC

* Remove the yaml files from the example scenarios

* update README with current status

* Add an alternative implementation that uses defer

* Forgot to add the old proc body

* slots-processing

* allow benching state_transition failures

* Add Attestations processing (workaround confutils bug:
- https://github.com/status-im/nim-confutils/issues/10
- https://github.com/status-im/nim-confutils/issues/11
- https://github.com/status-im/nim-confutils/issues/12

* Add CLI command in the readme

* Filter report and add notes about CPU cycles

* Report averages

* Add debugecho style time/cycle print

* Report when we skip BLS and state root verification

* Update to 0.9.3

* Generalize scenario parsing

* Support all block processing scenarios

* parallel bench runner PoC

* gitBetter load issues reporting (the load issues were invalid signature and expected to fail)
2019-12-20 17:14:43 +01:00
Dustin Brody 417f96213d fix genesis interop by fixing deposit signature handling; add TRACE-level compilation to CI 2019-12-20 15:26:48 +00:00
Dustin Brody ea4afd7454 mark 19 more 0.9.2-compatible entities as 0.9.3-compatible 2019-12-19 11:46:30 +00:00
Dustin Brody 350a6c071a mark a couple compatible dozen 0.9.2 functions/data structures as 0.9.3 2019-12-18 16:13:39 +00:00
Dustin Brody 0328cd49bf mark several more functions and data structures as 0.9.3 2019-12-18 10:00:17 +00:00
Dustin Brody 1610981be6 mark several more spec-referencing code/data parts as 0.9.3 2019-12-17 16:51:46 +00:00
Dustin Brody 27bf8908a9 mark a couple dozen functions and data structures as 0.9.3; fix TRACE-level compiling; stop downloading 0.9.2 test vectors; update cosmetic/display spec versions in test suite 2019-12-17 16:51:46 +00:00
Dustin Brody c824416f56 initial 0.9.3 spec update 2019-12-17 00:06:03 +02:00
Jacek Sieka a3145f0f5c cleanups
* move stuff out of beaconnode
* fix use-after-free in query
2019-12-10 14:47:52 +00:00
Jacek Sieka eea523a89e logging fixes
* don't log out-of-sync when there's no work to do
* small logging fixes
2019-12-10 10:18:30 +00:00
Dustin Brody c762c0232d render get_beacon_proposer_index(...) return value an option and remove overly aggressive assertion formerly causing crashes when missing validators (#617)
* render get_beacon_proposer_index(...) return value an option and remove overly aggressive assertion formerly causing crashes when missing validators

* follow addBlock(...) after refactoring

* [skip ci] fix typo
2019-12-04 11:49:59 +01:00
Dustin Brody e88e559caf render visible gradual validator emptying on catchup 2019-12-02 18:24:30 +02:00
Dustin Brody bde20436e8 ~50% speedup by fast-pathing Eth2Digest merkle hashing 2019-11-26 15:23:27 +00:00
Dustin Brody 1494bcc262 replace links to Apache and MIT licenses with HTTPS versions (#592) 2019-11-25 16:30:02 +01:00
Dustin Brody b82328b148 convert hash_tree_roots_prefix(...) to an iterator 2019-11-25 14:48:59 +00:00
Dustin Brody f6de87b80e remove O(n^2) eth1 deposit processing and scaffolding comparing it with O(n) deposit processing; re-enable now-viable-to-run genesis state initialization tests with around 16,384 validators 2019-11-25 14:48:59 +00:00
Dustin Brody fa2a703a17 switch more funcs and procs from global to module scope; create scaffolding for non-O(n^2) eth1 deposit processing with assertions for equivalent functionality; fix a few more shellcheck warnings 2019-11-25 14:48:59 +00:00
Dustin Brody a8de94ad05 add validator_leaving logging at debug chronicles level for all call paths to initiate_validator_exit(...) 2019-11-25 09:08:15 +00:00
Dustin Brody 8d9fdb0afd miscellanous cleanups and removing a few funcs/procs from unnecessary global scopes 2019-11-25 09:08:15 +00:00
Dustin Brody edfd65fd5d stop download 0.9.1 test vectors; finish 0.9.2 update, primarily via fixing preset constants; remove a couple 0.19.6 shims 2019-11-22 16:41:43 +00:00
Dustin Brody 41fa377638 mark all non-changed 0.9.1 items as 0.9.2 (#576) 2019-11-22 09:29:04 +01:00
Jacek Sieka 7c6d87051a
simplify validator api (#573)
* avoid passing whole state to validator api (easier when it goes
out-of-process)
* clean up some cruft
2019-11-21 10:57:59 +01:00
Dustin Brody c0f009908c
re-enable test_interop based on zcli with 0.9.1 specs and update initialize_beacon_state_from_eth1(...) to 0.9.1 (#569)
* re-enable test_interop based on zcli with 0.9.1 specs and update initialize_beacon_state_from_eth1(...) to 0.9.1

* switch many procs to funcs

* fix import os.nim instead; ospaths is deprecated [Deprecated] warnings
2019-11-21 09:15:10 +00:00
Dustin Brody 2749cb14a6 convert mostly-uncompiled verify_merkle_branch(...) calls to newer is_valid_merkle_branch(...) name; re-enable several lines of deposit testing code and outline roadmap for completion 2019-11-18 17:35:58 +00:00
Jacek Sieka 14712bbbdc
random warning fixes 2019-11-15 12:05:18 +01:00
Dustin Brody c08698bffe
scaffolding to remove last vestiges of get_committee_count(...), SHARD_COUNT, and rest of that 0.8.x infrastructure; this demonstrates equivalence between two, before discarding old code 2019-11-14 18:37:51 +01:00
Dustin Brody df1d00f13a remove UnusedImport spam (#549)
* remove UnusedImport spam

* re-add random import to beacon_node
2019-11-14 11:47:55 +01:00
Dustin Brody 5a54c823d8 update to 0.9.1: don't try to run removed tests; re-add 1.5 tests (one only in minimal); remove bls_verify_multiple(...) and AttestationDataAndCustodyBit; and update process_attester_slashing(...), get_indexed_attestation(...), and is_valid_indexed_attestation(...) 2019-11-13 13:49:38 +00:00
Dustin Brody d5ce142511 fix network sim finalization; remove get_attestation_data_slot(...); remove 2 more get_crosslink_committee(...) calls 2019-11-12 14:49:26 +00:00
Dustin Brody 568fa1dcb9 mark functions, constants, and data structures unchanged by 0.9.1 as such 2019-11-11 10:25:57 +00:00
Dustin Brody 949d735155 fix remaining test 0.9.0 test fixtures and switch back to non-transitiontest setup; remove get_committee_count(...); keep SHARD_COUNT consistent with MAX_COMMITTEES_PER_SLOT 2019-11-11 10:25:57 +00:00
Dustin Brody e984b2c932 mark is_valid_merkle_branch(...), process_justification_and_finalization(...), and process_rewards_and_penalties(...) as 0.9.0; update check_attestation(...) to 0.9.0; add 11 working test fixtures, having updated SSZ fixture directory to 0.9.0 2019-11-08 13:23:08 +00:00
Dustin Brody 3bb95f571f remove remaining Crosslink and CompactCommittee vestiges 2019-11-08 13:23:08 +00:00
Dustin Brody 61c2cf9415 rm get_compact_committees_root(...), Transfer, and references to 0.9-removed BeaconState.{active_index_roots,compact_committees_roots; temporarily disable genesis interop test pending 0.9 BeaconState 2019-11-08 13:23:08 +00:00
Dustin Brody 68654848cb update get_attesting_indices(...) to 0.9.0; remove Crosslink from AttestationData to update AttestationData to 0.9.0; rm get_winning_crosslink_and_attesting_indices(...) and get_crosslink_deltas(...) 2019-11-08 13:23:08 +00:00
Dustin Brody cb5454c4eb double-check that one can back-infer shard/epoch from slot/committee-index, in preparation for removing crosslink from AttestationData 2019-11-08 13:23:08 +00:00
Dustin Brody c2ebe482a5 temporarily disable SSZ tests, which fail when trying to read AttestationData or structures which embed it; implemenet get_committee_count_at_slot(...) and get_beacon_committee(...); express get_crosslink_committee(...) in terms of get_beacon_committee(...) and ensure that results are identical 2019-11-08 13:23:08 +00:00
Dustin Brody 0073555752
transition deposit operation fixtures to 0.9.0 (#515)
* transition deposit operations fixture to 0.9.0

* mark slash_validator(...) as 0.9.0

* switch remaining non-ref objects to ref objects to maybe avoid crashes in CI

* remove unused helpers/debug_state imports
2019-11-06 15:02:06 +00:00
Dustin Brody 63e621c27d
initial 0.9.0 spec sync (#509)
* rename compute_epoch_of_slot(...) to compute_epoch_at_slot(...)

* remove some unnecessary imports; remove some crosslink-related code and tests; complete renaming of compute_epoch_of_slot(...) to compute_epoch_at_slot(...)

* rm more transfer-related code and tests; rm more unnecessary strutils imports

* rm remaining unused imports

* remove useless get_empty_per_epoch_cache(...)/compute_start_slot_of_epoch(...) calls

* rename compute_start_slot_of_epoch(...) to compute_start_slot_at_epoch(...)

* rename ACTIVATION_EXIT_DELAY to MAX_SEED_LOOKAHEAD

* update domain types to 0.9.0

* mark AttesterSlashing, IndexedAttestation, AttestationDataAndCustodyBit, DepositData, BeaconBlockHeader, Fork, integer_squareroot(...), and process_voluntary_exit(...) as 0.9.0

* mark increase_balance(...), decrease_balance(...), get_block_root(...), CheckPoint, Deposit, PendingAttestation, HistoricalBatch, is_active_validator(...), and is_slashable_attestation_data(...) as 0.9.0

* mark compute_activation_exit_epoch(...), bls_verify(...), Validator, get_active_validator_indices(...), get_current_epoch(...), get_total_active_balance(...), and get_previous_epoch(...) as 0.9.0

* mark get_block_root_at_slot(...), ProposerSlashing, get_domain(...), VoluntaryExit, mainnet preset Gwei values, minimal preset max operations, process_block_header(...), and is_slashable_validator(...) as 0.9.0

* mark makeWithdrawalCredentials(...), get_validator_churn_limit(...), get_total_balance(...), is_valid_indexed_attestation(...), bls_aggregate_pubkeys(...), initial genesis value/constants, Attestation, get_randao_mix(...), mainnet preset max operations per block constants, minimal preset Gwei values and time parameters, process_eth1_data(...), get_shuffled_seq(...), compute_committee(...), and process_slots(...) as 0.9.0; partially update get_indexed_attestation(...) to 0.9.0 by removing crosslink refs and associated tests

* mark initiate_validator_exit(...), process_registry_updates(...), BeaconBlock, Eth1Data, compute_domain(...), process_randao(...), process_attester_slashing(...), get_base_reward(...), and process_slot(...) as 0.9.0
2019-10-30 19:41:19 +00:00
Dustin Brody 700e7d921b
update to beacon chain phase 0 spec version 0.8.4; no substantive changes (#494) 2019-10-24 13:36:36 +00:00
Jacek Sieka b7f9d9e4be fix BlsValue inits 2019-10-08 23:52:50 +09:00
Dustin Brody 255980c9f6 fix issue #367 and remove too-small range type usage for ValidatorIndex 2019-10-03 16:41:25 +03:00
Zahary Karadjov 806836714a Add an accidentally forgotten fix 2019-10-01 17:50:13 +03:00
Mamy Ratsimbazafy 6be2e24a8f
Attester slashing - split process_attester_slashing/processAttesterSlashings + tests (#424)
* attester slashing tests - pending #415

* split process_attester_slashing/processAttesterSlashings

* Add logs to attester_slashing

* deactivate bls tests for now (https://github.com/status-im/nim-beacon-chain/issues/429) and cherry-pick from 60f2437dd2
2019-09-10 18:03:06 -04:00
Yuriy Glukhov d4c480ff5f
Eth1 validators (#393) 2019-09-09 18:59:02 +03:00
Mamy Ratsimbazafy 3f446e6383
Attestations EF tests + Proper Real/Opaque BlsValue deserialization (#410)
* Add attestation unit test

* process_attestation doesn't throw exceptions

* Allow SSZ deserialization of both real and invalid signatures

* Add new process_attestation checks - pass all process_attestation tests

* Add sanity check for #361

* Fix SSZ testing after fromBytes/fromSSZBytes changes
2019-09-08 23:33:24 -04:00
Jacek Sieka 5f58af2112
fix ssz list, protocol names (#405) 2019-09-08 16:54:31 +02:00
Mamy Ratsimbazafy 82b9e008d6
Fix genesis blockheader zero signature (supercedes #395) (#400) 2019-09-07 20:56:24 -04:00
Dustin Brody 0c174036a5 update get_attesting_indices(...) to 0.8.3; mark IndexedAttestation, Attestation, PendingAttestation, and get_randao_mix(...) as 0.8.3; rm duplicate/dead code get_unslashed_attesting_indices(...) (#391) 2019-09-06 21:58:38 +02:00
Yuriy Glukhov e3bd4410d8
Revert "Eth1 validators" (#392)
This reverts commit bb8955bdbc.
2019-09-06 13:00:56 +03:00
Yuriy Glukhov bb8955bdbc Eth1 validators 2019-09-06 02:38:38 +02:00
Mamy Ratsimbazafy c11f37e550 V0.8.3 tests part 3 - sanity checks for slots and blocks (#375)
* Add sanity check for slot processing (also impacted by https://github.com/status-im/nim-beacon-chain/issues/373)

* use reportDiff also for all state tests vs EF

* initial sanity checks for blocks - workaround zero signature in block headers: https://github.com/status-im/nim-beacon-chain/issues/374

* Remove generic object variant compare commented code

* Add the one block state transition sanity checks

* generalize blocks test to multiple blocks

* simplify slots test runner

* Add official epoch transitions, sanity blocks, sanity slots to the test suite

* Fix index out-of-bounds in initiate_validator_exit - enable proposer slashings unittest
2019-09-03 18:02:21 +00:00
Mamy Ratsimbazafy 297e9079d4 V0.8.3 tests part2 - BLS + Shuffling + State transition epoch (#372)
* Update BLS fixtures to 0.8.3

* Bump fixtures with shuffling with minimal preset

* Update shuffling tests

* parseTest generic over file format (Json or SSZ)

* Initial crosslink parsing commit for debugging Nim crash

* Workaround https://github.com/status-im/nim-beacon-chain/issues/369

* Crosslink test works for minimal - https://github.com/status-im/nim-beacon-chain/issues/369 is back on mainnet

* Use ref objects to workaround https://github.com/status-im/nim-beacon-chain/issues/369

* Generalize state transition epoch test to all epoch tests

* Fix slashing (potential uint64 overflow in previous spec)

* Add a state debugging macro to deeply inspect the wrong fields

* make reportDiff visible

* Improve the debug state macro for containers
2019-09-03 03:12:09 +00:00
Jacek Sieka 6d798c821f log json errors when loading deposits 2019-09-02 22:14:18 +03:00
Jacek Sieka 9cfbdb5e16 more interop fixes
* fix eth1 interop block hash
* update initialize_beacon_state_from_eth1 to 0.8.3
2019-09-02 22:14:18 +03:00
Jacek Sieka 7b73b40bab interop updates
* add interop launcher scripts
* stick validator_keygen into beacon_node
* fix lmd ghost slot number on missing block
* use mocked eth1data when producing blocks
* use bls public key method for withdrawal credentials
* fix deposit domain
* prefer lowercase for a bunch of toHex
* build simulation binary in data folder to avoid data types confusion
2019-09-02 22:14:18 +03:00
Mamy Ratsimbazafy ca4f29caca v0.8.1 tests refactor (#326)
* Introduce new mocking proc to replace:
- makeFakeValidatorPrivKey
- hackPrivKey
- getNextBeaconProposerIndex
- addBlock
- makeBlock

* Add comments on datastructure unsynced with the spec

* Add merkle tree constructor and initial mocking for deposits (missing merkle proofs)

* [Mock] Implement sparse merkle tree and merkle proof builder

* [Mocking] Genesis deposits

* Add compact_committees_roots init + mock genesis state

* [Tests] Add first deposit test using the new mocking procedures

* [Tests -deposits] add at and over 32 ETH deposit tests

* [Tests - deposits] Add test for validator top-up

* [Tests -deposits] Mention the TODO to test for invalid conditions

* [Tests] Add stub to test "is_valid_genesis_state"

* [Merkle proofs] Implement round-trip checks

* Deactivate roundtrips test

* SSZ - use EF convention for hash_tree_root / hashTreeRoot

* [Tests - Attestation] Attestation mocking + initial test

* Add mocking + 3 new tests for valid attestations + mention future invalid attestation tests

* Add crosslinks test (1 failing to attestations in block being duplicated in state transition)

* Single attestation crosslink test - workaround https://github.com/status-im/nim-beacon-chain/issues/361

* Add test for failed crosslink penalty

* Rebase fixes + add refactored tests to test suite

* justif-finalization helpers first batch

* Add 234 finalization tests

* Fix justif test, Rule I 234 finalization does not happen with sufficient support.
(Also unittest check template does not fail properly in some cases)

* Add tests for all finalization rules

* Properly delete nim-byteutils following c91727e7e5 (diff-7c3613dba5171cb6027c67835dd3b9d4)

* use digest helper for deposit root
2019-08-28 12:07:00 +00:00
Dustin Brody 8665efbbd1 bump beacon chain versions to 0.8.3, which has no relevant changes in this regard (#358) 2019-08-26 11:09:47 +02:00
Dustin Brody cc0429a643
slightly update/rename verify_merkle_branch(...) to 0.8.2 is_valid_merkle_branch(...); mark check_attestation(...), get_active_validator_indices(...), get_seed(...), and BeaconBlockBody as 0.8.2; update compute_domain(...) 0.8.2 by reversing field order (#356) 2019-08-26 07:57:09 +00:00
Jacek Sieka 78f7a4a505 restore network sim finalization
* relax attestation validation when attestation is incoming but make it
more strict when adding to block
* share attestation validation logic between attestation pool and state
transition
* remove a bunch of redundant logging
* fix potential underflow in attestation delay checking
* fix committee used for attestation shard selection when attesting
* fix attestation data construction
2019-08-19 21:05:54 +03:00
Jacek Sieka e639cd95c7
humaneXNum -> shortLog 2019-08-15 18:01:55 +02:00
Zahary Karadjov 4daeebbf7e
Fix the build when json logging is enabled 2019-08-14 17:38:39 +02:00
Dustin Brody 3e587d4667
Justification/finalization amelioration using testnet1 shard count and epoch length (#331)
* working justification, 1 node, 64 validators

* closer to tn1 params: 256 validators

* more debugging output; switch to minimum test case

* working justification and finalization in local network simulation

* fix currently incorrect state transition/attestation test assumption
2019-08-14 08:56:32 +00:00
Dustin Brody 6704749eb0 mark unchanged 0.8.1 as 0.8.2 spec aspects (#336) 2019-08-09 14:08:00 +02:00
Zahary Karadjov 398ea55801
Implement the latest SSZ specification and integrate the official SSZ test suite 2019-08-05 03:18:48 +03:00
Yuriy Glukhov 958e4cd3b4 Getting genesis through web3 2019-07-31 16:39:26 +03:00
Dustin Brody 712038fe21
mark applicable 0.8.0-compliant functions and data structures as 0.8.1 and update a couple of other comments (#321) 2019-07-15 12:57:18 +00:00
Dustin Brody dfa062db7d 0.8.0, continued (#315)
* rename get_genesis_beacon_state(...) to initialize_beacon_state_from_eth1(...); remove unused vestiges get_temporary_block_header(...) and get_empty_block(...); partly align initialize_beacon_state_from_eth1(...) with 0.8.0 version; implement CompactCommittee object type; update process_final_updates(...) to 0.8.0

* mark get_shard_delta(...) as 0.8.0

* mark get_total_active_balance(...) and epoch transition helper functions as 0.8.0

* move FORK_CHOICE_BALANCE_INCREMENT, not in spec anymore, to sole user in fork_choice
2019-07-10 14:23:02 +02:00
Dustin Brody 68be98cafa
More 0.8.0 updates (#311)
* replace BeaconState.finalized_{epoch,root} with BeaconState.finalized_checkpoint; rename get_delayed_activation_exit_epoch(...) to compute_activation_exit_epoch(...) and mark as 0.8.0; update get_churn_limit(...)/get_validator_churn_limit(...) to 0.8.0; update process_registry_updates(...) to 0.8.0

* update process_crosslinks(...) to 0.8.0; mark compute_start_slot_of_epoch(...) and get_committee_count(...) as 0.8.0

* mark Fork, is_slashable_validator(...), and get_beacon_proposer_index(...) as 0.8.0

* rename LATEST_SLASHED_EXIT_LENGTH to EPOCHS_PER_SLASHINGS_VECTOR; update process_slashings(...) to 0.8.0; remove pointless type conversion warning in get_previous_epoch(...)

* convert remaining references to finalized_epoch to finalized_checkpoint.epoch

* update slash_validator(...) to 0.8.0; mark inital value, Gwei, and time constants as 0.8.0; mark hash(...) and processBlockHeader(...) as 0.8.0

* rename WHISTLEBLOWING_REWARD_QUOTIENT to WHISTLEBLOWER_REWARD_QUOTIENT; rename LATEST_ACTIVE_INDEX_ROOTS_LENGTH to EPOCHS_PER_HISTORICAL_VECTOR (randao will also get merged into this); remove get_active_index_root(...); mark time parameter, signature domain types, and max operations per block constants as 0.8.0; update rewards and penalties constants to 0.8.0

* update is_valid_indexed_attestation(...) to 0.8.0; mark process_slot(...) as 0.8.0

* replace BeaconState.{current,previous}_justified_{epoch,root} with BeaconState.{current,previous}_justified_checkpoint
2019-07-05 08:30:05 +00:00
Dustin Brody 5198930503 add start_epoch consistent with rest of crosslink 2019-07-03 10:41:42 +03:00
Dustin Brody 8a5e5334d6 re-implement attestation checking in 0.8.0 form and rename relevant function from checkAttestation(...) to spec's process_attestation(...), though there's a slight difference in how they're factored; remove a couple of pointless type conversion warnings; rename validate_indexed_attestation(...) to is_valid_indexed_attestation(...); remove verify_bitfield(...) not in 0.8.0; update AttestationData to 0.8.0 by using Checkpoint data structure; rename MAX_CROSSLINK_EPOCHS to MAX_EPOCHS_PER_CROSSLINK 2019-07-03 10:41:42 +03:00
Dustin Brody d7905351eb update get_crosslink_committee(...) to 0.8.0 and fix https://github.com/status-im/nim-beacon-chain/issues/307 2019-07-02 19:52:39 +03:00
Dustin Brody 95dea4c165 mark get_attestation_data_slot(...), ProposerSlashing, AttesterSlashing, DepositData, BeaconBlock, Eth1Data, AttestationDataAndCustodyBit, and is_slashable_attestation_data(...) as 0.8.0; add Checkpoint object type 2019-07-01 14:44:57 +03:00
Dustin Brody fa39eb82c4 do most of 0.8.0 BeaconState updating, in particular field renamings, re-orderings, etc; rm now-unnecessary genesis block creation aspects, towards the near-stub version in current spec as mostfields are 0/default/etc values 2019-07-01 14:44:57 +03:00
Dustin Brody fb601bd5b8 rename BeaconState.validator_registry to BeaconState.validators 2019-07-01 14:44:57 +03:00
Dustin Brody 2ba14bc9af update get_previous_epoch(...) to 0.8.0 and mark compute_committee(...) as 0.8.0 2019-07-01 14:44:57 +03:00
Dustin Brody 607a7de5eb rename get_epoch_start_shard(...) to 0.8.0 get_start_shard(...) 2019-07-01 14:44:57 +03:00
Dustin Brody e4321dc4ed Initial frozen phase 0/v0.8.0 spec (#305)
* rename slot_to_epoch(...) to compute_epoch_of_slot(...)

* rename aggregation_bitfield/custody_bitfield fields to aggregation_bits/custody_bits; rename convert_to_indexed(...) to get_indexed_attestation(...); mark BeaconBlockHeader as 0.8.0; update minimal preset MIN_ATTESTATION_INCLUSION_DELAY/time parameters in general to 0.8

* fix beacon node compilation; it used slightly different capitalizations of slot_to_epoch/slotToEpoch

* mark integer_squareroot(...), Deposit, VoluntaryExit, and Transfer as 0.8.0; rename get_epoch_start_slot(...) to compute_start_slot_of_epoch(...)

* add new Domain alias for uint64; rename bls_domain(...) to compute_domain(...), MAX_INDICES_PER_ATTESTATION to MAX_VALIDATORS_PER_COMMITTEE, and SignatureDomain to DomainType; update get_domain(...) to 0.8.0

* update/mark initiate_validator_exit(...) and Crosslink as 0.8.0; rename BeaconState.latest_block_roots -> BeaconState.block_roots; mark HistoricalBatch as 0.8.0
2019-07-01 09:53:42 +02:00
Dustin Brody 7356905f95
remove get_epoch_start_slot(...) kludge and update to work exactly as 0.7.1 specifies; remove unused get_attestation_participants_cached(...); update AttestationData to 0.7.1 (being the last data structure, I believe, remaining for such; remove potentially spurious/certainly not-in-spec assertion from compute_committee(...); fix state sim to work with new get_epoch_start_slot/AttestationData/etc setup where it can't stuff all shards' attestations from same slot in the same MIN_ATTESTATION_INCLUSION_DELAY rotating/circular buffer of seq[Attestation]s without more involved shuffling of shard/slot calculation order; fix attestation pool testing to be consistent with get_epoch_start_slot(...) (#302) 2019-06-29 09:17:24 +00:00
Dustin Brody 43a9e904cf
replace get_attestation_slot(...) with 0.7.1 get_attestation_data_slot(...) and update all callers; update AttestationData to be almost 0.7.1, except nonextistent slot field; update Crosslink and get_winning_crosslink_and_attesting_indices(...) to 0.7.1 (#297) 2019-06-25 11:17:08 +00:00
Dustin Brody c7e06374f4
Remove get_crosslink_committees_at_slot and fix research/state_sim (#291)
* migrate attestation pool tests from get_crosslink_committees_at_slot(...) to get_crosslink_committee(...)

* rm obsolete, unused get_crosslink_committees_at_slot_cached(...) and migrate tests/test_state_transition from get_crosslink_committees_at_slot(...) to get_crosslink_committee(...)

* migrate tests/testutil from get_crosslink_committees_at_slot(...) to get_crosslink_committee(...)

* use more pervasive caching infrastructure, initially of compute_committee; remove buggy (and per-index) shuffling to fix research/state_sim, which was noticing validators on multiple shard committees; rm now-unused specific get_attesting_balance_cached

* add some get_active_validator_index caching

* rm obsolete/unused get_attesting_indices_cached

* rm get_crosslink_committees_at_slot(...)

* some more 0.7 changes -- some of which can't be completed pending some data structure updates -- and shard handling changes to offset with epoch start shards
2019-06-24 09:21:56 +00:00
Dustin Brody 13381c8fea
update everything https://github.com/ethereum/eth2.0-specs/releases/tag/v0.7.1 leaves unchanged from v0.7.0 to v0.7.1 (#290) 2019-06-18 15:34:30 +00:00
Dustin Brody e9180c490e
more 0.7.0 updates (#287)
* update Deposit and BeaconBlockHeader to 0.7.0

* update get_total_balance(...) and get_base_reward(...) to 0.7.0

* rename initial value constant BLS_WITHDRAWAL_PREFIX_BYTE to BLS_WITHDRAWAL_PREFIX

* mark minimal presets as entirely migrated to 0.7.0; update process_rewards_and_penalties(...) to 0.7.0

* specify that targeting 0.7.0; adopt 0.7.0 refactoring of bls_domain/get_domain to update get_domain(...) to 0.7.0; mark that mainnet constant presets and block header processing are 0.7.0-consistent; update process_justification_and_finalization to 0.7.0 to fix potential underflow errors

* address mratsim's comment to copy whole spec comment
2019-06-14 16:21:04 +00:00
Dustin Brody 2a6e64d3f9
0.7.0 updates including semantic changes (#285)
* update get_previous_epoch(...)

* update BeaconBlock to 0.7.0 by renaming previous_block_root field to parent_root
2019-06-14 13:50:47 +00:00
Dustin Brody e2496567b3 More safe 0.7.0 updates (#283)
* complete marking unchanged-in-0.7.0 beaconstate functions

* complete marking unchanged-in-0.7.0 spec refs functions in helpers.nim and time.nim: get_randao_mix(...), bytes_to_int(...), and description of beacon chain processing apropos time

* complete remaining feasible 0.7.0 updates which lack externally visible effect
2019-06-14 09:50:14 +02:00
Dustin Brody 6357bc2a98 0.7.0 updates which don't change semantics; mostly comment changes (#281)
* 0.7.0 updates which don't change semantics; mostly comment changes

* mark get_attesting_indices, bls_verify, ProposerSlashing, BeaconBlockBody, deposit contract constants, state list length constants, compute_committee, get_crosslink_committee, is_slashable_attestation_data, processAttesterSlashings as 0.7.0; rename BASE_REWARD_QUOTIENT to BASE_REWARD_FACTOR

* complete marking unchanged-in-0.7.0 datatypes as such; a few more have trivial field renamings, etc
2019-06-13 09:40:58 +02:00
Dustin Brody d400650eeb
0.6.3 updates (#279)
* update IndexedAttestation, verify_slashable_attestation/verify_indexed_attestation, and attester slashing processing to 0.6.3

* rm debug scaffolding

* rename Attestation.aggregate_signature -> Attestation.signature; convert various references to AttestationData.slot to get_attestation_slot; implement convert_to_indexed; update checkAttestation and processAttestations to 0.6.3; remove spurious assertion in beacon node related to invalid attestations

* replace 0.5 get_winning_root_and_participants with 0.6 get_winning_crosslink_and_attesting_indices; update process_crosslinks to 0.6.3

* mark both remaining 0.6.0 spec implementations as 0.6.3

* clear out all remaining spec version 0.6.1 refs

* GENESIS_SLOT and GENESIS_EPOCH are 0

* rm 0.5 get_attestation_participants in favor of 0.6 get_attesting_indices

* address mratsim's comment

* time can be equal to genesis

* fix invalid block assertions; those were essentially spurious

* allow toBeaconTime to handle time before genesis (in accordance with now(...) which states time can exist before GENESIS)
2019-06-12 07:48:49 +00:00
Dustin Brody 252819a84d
bump everything that's not one of the 3 or so specific changes in 0.6.3 from 0.6.2 to 0.6.3 if already at 0.6.2 (#277) 2019-06-05 10:25:30 +00:00
Dustin Brody 2e1515b107
0.6.2 updates (#275)
* update process_justification_and_finalization to 0.6.2; mark AttesterSlashing as 0.6.2

* replace get_effective_balance(...) with state.validator_registry[idx].effective_balance; rm get_effective_balance, process_ejections, should_update_validator_registry, update_validator_registry, and update_registry_and_shuffling_data; update get_total_balance to 0.6.2; implement process_registry_updates

* rm exit_validator; implement is_slashable_attestation_data; partly update processAttesterSlashings

* mark HistoricalBatch and Eth1Data as 0.6.2; implement get_shard_delta(...); replace 0.5 finish_epoch_update with 0.6 process_final_updates

* mark increase_balance, decrease_balance, get_delayed_activation_exit_epoch, bls_aggregate_pubkeys, bls_verify_multiple, Attestation, Transfer, slot_to_epoch, Crosslink, get_current_epoch, int_to_bytes*, various constants, processEth1Data, processTransfers, and verifyStateRoot as 0.6.2; rm is_double_vote and is_surround_vote

* mark get_bitfield_bit, verify_bitfield, ProposerSlashing, DepositData, VoluntaryExit, PendingAttestation, Fork, integer_squareroot, get_epoch_start_slot, is_active_validator, generate_seed, some constants to 0.6.2; rename MIN_PENALTY_QUOTIENT to MIN_SLASHING_PENALTY_QUOTIENT

* rm get_previous_total_balance, get_current_epoch_boundary_attestations, get_previous_epoch_boundary_attestations, and get_previous_epoch_matching_head_attestations

* update BeaconState to 0.6.2; simplify legacy get_crosslink_committees_at_slot infrastructure a bit by noting that registry_change is always false; reimplment 0.5 get_crosslink_committees_at_slot in terms of 0.6 get_crosslink_committee

* mark process_deposit(...), get_block_root_at_slot(...), get_block_root(...), Deposit, BeaconBlockHeader, BeaconBlockBody, hash(...), get_active_index_root(...), various constants, get_shard_delta(...), get_epoch_start_shard(...), get_crosslink_committee(...), processRandao(...), processVoluntaryExits(...), cacheState(...) as 0.6.2

* rm removed-since-0.5 split(...), is_power_of_2(...), get_shuffling(...); rm 0.5 versions of get_active_validator_indices and get_epoch_committee_count; add a few tests for integer_squareroot

* mark bytes_to_int(...) and advanceState(...) as 0.6.2

* rm 0.5 get_attesting_indices; update get_attesting_balance to 0.6.2

* another tiny commit to poke AppVeyor to maybe not timeout at connecting to GitHub partway through CI: mark get_churn_limit(...), initiate_validator_exit(...), and Validator as 0.6.2

* mark get_attestation_slot(...), AttestationDataAndCustodyBit, and BeaconBlock as 0.6.2
2019-06-03 10:31:04 +00:00
Dustin Brody 6a47ca3c46
More 0.6.1 updates (#274)
* implement get_churn_limit from 0.6.1; update initiate_validator_exit and get_previous_epoch to 0.6.1; remove obsoleted/non-spec reduce_balance, replaced by decrease_balance; mark BeaconBlock as 0.6.1

* rename get_block_root to get_block_root_at_slot and introduce epoch-oriented get_block_root; implement get_attestation_slot, get_matching_source_attestations, get_matching_target_attestations, get_matching_head_attestations, 0.6.1 get_attesting_indices, get_unslashed_attesting_indices, get_attestation_deltas, process_rewards_and_penalties; rm get_inactivity_penalty

* update Validator and processVoluntaryExits to 0.6.1; rm obsolete inclusion_slots/inclusion_distances

* rm removed activate_validator; mark DepositData, misc values, Gwei values, Randao processing, state caching as 0.6.1; update GENESIS_SLOT to 64, since it doesn't quite yet work at 0

* mark BeaconBlockHeader as 0.6.1; update BeaconBlockBody to 0.6.1; rename WHISTLEBLOWER_REWARD_QUOTIENT to WHISTLEBLOWING_REWARD_QUOTIENT; update slash_validator to 0.6.1

* implement 0.6.2 is_slashable_validator; update processProposerSlashings to 0.6.2; mark state caching as 0.6.2

* mark get_total_active_balance and process_slashings as 0.6.2; update get_base_reward to 0.6.2

* rm prepare_validator_for_withdrawal, process_exit_queue; mark mainnet misc constants as 0.6.2; update process block header to 0.6.2

* address mratsim's code review comment
2019-05-29 10:08:56 +00:00
Dustin Brody 46cfce652d
initial new get_beacon_proposer_index and dependencies (get_shuffled_index, get_shard_delta, get_epoch_start_shard, get_crosslink_committee) (#266)
* initial new get_beacon_proposer_index and dependencies (get_shuffled_index, get_shard_delta, get_epoch_start_shard, get_crosslink_committee)

* more 0.6.1 updates

* clean up debugEchos and fix validator keygen to create correct indices
2019-05-23 11:13:02 +00:00
Dustin Brody d977c96ae1 0.6.1 - signing_root, deposits, validator indices and committee count (#261)
* signed_root -> signing_root

* implement new process_deposit; update timing parameters to 0.6.1; update Deposit to 0.6.1 and remove DepositInput

* update get_active_validator_indices and get_epoch_committee_count to 0.6.1; rm get_current_epoch_committee_count, get_previous_epoch_committee_count, and get_next_epoch_committee_count; bump state_sim default validator count a bit more

* re-introduce 0.5.1-ish get_active_validator_indices, get_epoch_committee_count as scaffolding for still-0.5.1ish shuffling
2019-05-09 14:27:37 +02:00
Dustin Brody 46b4154ce8 begin 0.6.0 (#256)
* begin 0.6.0: new get_domain/increase_balance/reduce_balance, BeaconState.validator_balances -> BeaconState.balances, some renamed constants, transaction processing changes, SlashableAttestation field name changes, 0.6.0 get_beacon_proposer_index always uses given state's slot, update tests subrepo

* mark get_bitfield_bit/bls_verify_multiple/stat-list-lengths/is_active_validator/is_surround_vote/slot_to_epoch/int_to_bytes/etc as unchanged in 0.6.0; rm Eth1DataVote/maybe_reset_eth1_period and thus adjust expected tree hash test results

* mark verify_bitfield/bls_verify/deposit-contract/VoluntaryExit/PendingAttestation/Historicalbatch/Fork as 0.6.0; update DOMAIN_BEACON_BLOCK to DOMAIN_BEACON_PROPOSER

* update Crosslink to 0.6.0 (also requires tree hashing test result change, so isolate in individual commit)

* mark verify_merkle_branch/get_delayed_activation_exit_epoch/ProposerSlashing/Attestation/AttestationDataAndCustodyBit/hash/integer_squareroot/get_epoch_start_slot/is_double_vote/get_randao_mix/generate_seed as 0.6.0; update reward and penalty quotients; SlashableAttestation -> IndexedAttestation; rm get_fork_version; ATTESTATION_INCLUSION_REWARD_QUOTIENT -> PROPOSER_REWARD_QUOTIENT
2019-04-29 10:48:30 -06:00
Jacek Sieka 9a2f3176f7 guard node.state (#237)
* fix node.state being used unsafely across async callbacks (fixes #230)
* node.state -> node.stateCache
* fix attestation validation not to use asserts
* re-enable attestation pool tests
* prepare for resizing validator registry
2019-04-06 09:46:07 +02:00
Dustin Brody c53de3e550 Caching updates/refactoring & state_sim defaults updates (#235)
* rm now-superceded shuffling cache (shuffling is only called from get_crosslinks), which was badly architected due to trying to exist in state; rm one more vestige of previous light-client regime (one more to go, from datatypes)

* fix wrong shuffling list size (active validator size, not validator size) to make consistent with 0.5.1 (will be inconsistent with testnet0); fix typo and change defaults in state_sim

* doAssert a couple of constant relationships necessary to avoid underflow; rm non-spec, unused helper function get_new_recent_block_roots

* refactor separate crosslink_committee_cache and winning_root_participants_cache(s) into StateData object; remove last vestige of previous shuffling cache

* separate out caching parts of StateData to new StateCache object
2019-04-05 08:18:13 -06:00
Jacek Sieka 605dd0a0e9 Some speedups (#226)
* ssz: avoid memory allocations

* a bit fishy with the 32-item stack.. this should be a smallvector

* digest: avoid another burnmem

* avoid a few allocations by using iterator
2019-04-03 15:46:22 +00:00
Jacek Sieka 201b079fe3 avoid bad blocks by filtering included attestations
* fix epoch violation during initial attestation validation
* fill in attestation fields (finalizes again!)
2019-03-28 21:10:08 +02:00
Dustin Brody ced4dbe859 4x speedup on epoch processing for 64 validators 2019-03-28 10:40:20 +02:00
Jacek Sieka 81c66fbce0
use signed_root as canonical block root (#211)
* no need to pass prev_block_root when updating state
* some Slot fixes
* fix `hash_tree_root` for Slot, Epoch
* detect missing hash_tree_root type support
* remove some <0.5 state checks
2019-03-26 19:32:35 -06:00
Jacek Sieka 1b0e67c88c
ssz: update to 0.5.1:ish (#202)
* ssz: update to 0.5.1:ish
* slightly fewer seq allocations
* still a lot of potential for optimization
* fixes #174
* ssz: avoid reallocating leaves (logN merkle impl)
2019-03-25 10:46:31 -06:00
Dustin Brody 6b1527ccd5 Almost final 0.5.1 spec updates (#201)
* update processExits to 0.5.1

* mark process_deposit, get_empty_block as 0.5.1; update get_genesis_beacon_state to 0.5.1; implement processDeposits

* hotfix via https://github.com/ethereum/eth2.0-specs/pull/821
2019-03-24 12:21:23 -06:00
Dustin Brody 9f55b4646c
More 0.5.1 spec updates (#195)
* rm gone-in-0.5.0 Proposal, verifyBlockSignature, and slot check which moved to spec function processBlockHeader

* mark get_attestation_participants and get_epoch_committee_count as 0.5.1; finish updating processAttestations to 0.5.1; add kludgy workaround for bug relating to get_winning_roots_etc using crosslink_data_root as index when we have that as ZERO_HASH for all leading to it confusing attesters on different shards; rm BeaconState.latest_attestations, which splits into previous_epoch_attestations and current_epoch_attestations

* Fix CI due to removed latest_attestations field
2019-03-22 18:33:12 +00:00
Dustin Brody 94c3b005fa
https://github.com/ethereum/eth2.0-specs/pull/816 hotfix (#194) 2019-03-21 16:02:50 +00:00
Dustin Brody 1bb2859aca
implement bulk of 0.5.0 state transition function (#192) 2019-03-21 15:31:36 +00:00
Jacek Sieka ee89ef1c79
cleanups (#189)
* add simple bitfield type (fixes #19)
* fix bit endianess to match spec
* consolidate attestation and block logging
* cruft cleanup
* run optimizer on tests, speeds up build
2019-03-20 14:01:48 -06:00
Dustin Brody 13c7f7fcec begin refactoring 0.5.0 epoch transition (#185)
* begin refactoring 0.5.0 epoch transition at justification and finalization

* update processTransfers to 0.5.0 spec

* update get_crosslink_committees_at_slot to 0.5.0 except for noted workaround for pre-0.5.0 spec bug related to epoch boundary conditions, to be flipped in more coordinated way with other epoch processing changes

* mark ejection processing as 0.5.0

* 0.5.0 epoch update finalization

* rm BEACON_CHAIN_SHARD_NUMBER to complete updating miscellaneous constants to 0.5.0

* mark verify_slashable_attestation as 0.5.0

* update process_attester_slashing to 0.5.0

* update process_slashings(...) to 0.5.0 and mark process_exit_queue as 0.5.0

* refactor epoch processing by implementing 0.5.0's update_registry_and_shuffling_data(...)
2019-03-19 20:19:46 -06:00
Dustin Brody 2bc632c29b
AttestationData.justified epoch -> AttestationData.source_epoch and AttestationData.latest_crosslink -> AttestationData.previous_crosslink; implement verify_merkle_branch and update process_deposit to use it; rm BeaconState.batched_block_roots; Transfer.{from, to} -> Transfer.{sender, recipient}; update Fork data type and code to use bytes4; implement int_to_bytes8 (close to metaprogramming time); adjust signed_root and all callers to use 0.5.0 calling signature; processDepositRoot -> processEth1Data (#182) 2019-03-18 15:42:42 +00:00
Dustin Brody 798197879b
Revert "AttestationData.justified epoch -> AttestationData.source_epoch and AttestationData.latest_crosslink -> AttestationData.previous_crosslink; implement verify_merkle_branch and update process_deposit to use it; rm BeaconState.batched_block_roots; Transfer.{from, to} -> Transfer.{sender, recipient}; update Fork data type and code to use bytes4; implement int_to_bytes8 (close to metaprogramming time); adjust signed_root and all callers to use 0.5.0 calling signature; processDepositRoot -> processEth1Data"
This reverts commit 18f1d000a1.
2019-03-17 15:24:50 -07:00
Dustin Brody 18f1d000a1
AttestationData.justified epoch -> AttestationData.source_epoch and AttestationData.latest_crosslink -> AttestationData.previous_crosslink; implement verify_merkle_branch and update process_deposit to use it; rm BeaconState.batched_block_roots; Transfer.{from, to} -> Transfer.{sender, recipient}; update Fork data type and code to use bytes4; implement int_to_bytes8 (close to metaprogramming time); adjust signed_root and all callers to use 0.5.0 calling signature; processDepositRoot -> processEth1Data 2019-03-17 15:17:40 -07:00
Dustin Brody f36c2d86dc begin 0.5.0 spec update (#179)
* begin 0.5.0 spec update: parent_root -> previous_block_root, BeaconState.justified_epoch -> BeaconState.current_justified_epoch, DOMAIN_PROPOSAL -> DOMAIN_BEACON_BLOCK, temporarily rename BeaconBlockHeader to BeaconBlockHeaderRLP to allow for gradual re-merging without disrupting RLP; mark a few unchanged functions and data types, implement get_temporary_block_header/get_empty_block/should_update_validator_registry/processBlockHeader/cacheState; update a few others

* a dozen or so more trivial mostly comment changes, finding more unchanged parts of 0.5.0

* several more trivial changes; goal is to reduce noise around the upcoming substantial changes (epoch processing, etc)
2019-03-16 13:52:37 -06:00
Jacek Sieka 1cb8ae9004
state processing fixes (#177)
* remove some redundant state updates
* when attesting late, use correct state / head
* don't send out obsolete attestations
* don't propose obsolete blocks
* remove some more resundant state updates :)
* simplify block logging (experimental)
* document fork choice division
* fix some Slot / Epoch conversion warnings
2019-03-14 07:33:56 -06:00
Ștefan Talpalaru de295619be
assert() -> doAssert() 2019-03-14 00:04:43 +01:00
Zahary Karadjov 562eafe124 Finish the Spec/Epoch types refactoring 2019-03-13 23:23:01 +02:00
Dustin Brody 1479bae22f implement distinct Epoch type and continue 0.4.0 spec update (#173)
* implement distinct Epoch type
* update two more spec 0.4.0 markers, leaving just deposit processing
2019-03-12 19:46:44 -06:00
Dustin Brody 50a0948bc0 Mostly convert to using distinct Slot type (#172) 2019-03-12 16:21:32 -06:00
Dustin Brody 434ba5727f finish all but 3 items for 0.4.0 spec update (#167) 2019-03-12 10:59:43 +01:00