Commit Graph

1092 Commits

Author SHA1 Message Date
Jacek Sieka 9244ae7a38 more speedups
* evaluate block attestations under the epochref of the block - this is
what the state transition function does
* avoid copying attestation seq unnecessarily
* avoid unnecessary hashset for unslashed indices
2020-08-19 14:51:04 +03:00
Jacek Sieka 7de05efaaf small perf fixes
* don't sort shuffled_validator_indices, just get them directly with
iteration
* grab full epoch of proposer indices while we have the data available -
they'll get cached and reused
* avoid computing active validator set when not used for logging
2020-08-19 14:51:04 +03:00
Zahary Karadjov 2c19e3f8cd
[skip ci] Use GOSSIP_MAX_SIZE when snappy decoding in the inspector as well; Bumps 2020-08-19 14:33:52 +03:00
Jacek Sieka 46c94a18ba rework epoch cache referencing
* collect all epochrefs in specific blocks to make them easier to find
and to avoid lots of small seqs
* reuse validator key databases more aggressively by comparing keys
* make state cache available from within `withState`
* make epochRef available from within onBlockAdded callback
* integrate getEpochInfo into block resolution and epoch ref logic such
that epochrefs are created when blocks are added to pool or lazily when
needed by a getEpochRef
* fill state cache better from EpochRef, speeding up replay and
validation
* store epochRef in specific blocks to make them easier to find and
reuse
* fix database corruption when state is saved while replaying quarantine
* replay slots fully from block pool before processing state
* compare bls values more smartly
* store epoch state without block applied in database - it's recommended
to resync the node!

this branch will drastically speed up processing in times of long
non-finality, as well as cut memory usage by 10x during the recent
medalla madness.
2020-08-19 10:09:06 +03:00
tersec 612881b95d
refactor topic (un)subscribing/validating to collate each (#1510)
* refactor topic (un)subscribing/validating to collate each

* fix comment

* tweak comment
2020-08-17 14:07:29 +02:00
tersec f34eddb6e9
fix get_unslashed_attesting_indices() and add official EF rewards tests for it (#1514) 2020-08-17 01:09:27 +00:00
tersec bc6eefe31e
add --enable-logtrace argument to launch_local_testnet (#1502)
* add --enable-logtrace argument to launch_local_testnet

* scan for all available logfiles

* remove specific filename references

* update v0.11.3 spec ref to v0.12.2
2020-08-16 11:12:19 +02:00
Mamy Ratsimbazafy 454b9d0724
Bump nim-blscurve (#1491)
* Bump BLSCurve

* Use unified aggregation API

* use new blscurve with unified aggregate API

* bump

* fix toRaw

* replace state_sim combine with AggregateSignature

* Fix 32-bit

* Fix 32-bit for real and test deactivating ccache for fno-tree-lopp-vectorize flag

* change compilation switches to narrow down Linux issue

* Use -fno-tree-vectorize to disable both tree-loop-vectorize and tree-slp-vectorize

* blscurve now disables both Loop and SLP vectorization

* Add tests for the miracl/milagro fallback

* Travis has max log size of 4MB

* Test with Miracl in the finalization test

* fix state_sim log level

* Coment out the slow fallback tests
2020-08-15 19:33:58 +02:00
tersec 611c5097cc
use cache in process_voluntary_exit() (#1507) 2020-08-14 12:42:59 +00:00
Jacek Sieka a1bd44f4b0
small spec cleanups (#1501)
* clean up logging a bit
* return error on indexed attestation check
2020-08-13 13:47:06 +00:00
tersec ab34584f23
initial dynamic subscribe/unsubscribe for attestations to/from subnets (#1462)
* initial dynamic subscribe/unsubscribe for attestations to/from subnets

* implement random stability subnet and clean up

* switch from HashSet[uint64] to set[uint8]

* refactor subnet logic out from beacon_node and actual (un)subscribing

* only try to subscribe to marginally different subnets

* add assertions

* maintain ENR subnets

* assert that beacon_node and eth2_network have consistent view of subscribed subnets

* disable actual cycling
2020-08-12 17:48:31 +00:00
tersec af3355e0f8
create local testnet mode for eth2_network (#1494) 2020-08-12 14:16:59 +00:00
tersec 22c1ef5a8d
split subscribe into non-validating subscribe and addValidator (#1485)
* split subscribe into non-validating subscribe and addValidator

* stop exporting get_committee_assignments
2020-08-11 15:08:44 +00:00
Jacek Sieka 2b4526e743
bls: avoid exception flow on cache miss (#1479) 2020-08-10 14:51:23 +00:00
Jacek Sieka 84a501d1ff
remove one cache, add another (#1449)
* remove one cache, add another

This cache removes the need for rewinding in most attestation validation
flow since the attestations come from one of two epochs and must be
targetting a viable block.

Additionally, it also removes all state caches which are less likely to
be used over-all - more metrics are needed to track the rewinding.

On risk is that when chains don't finalize, we'll have lots of epochrefs
in memory meaning lots of validator key databases, most being exactly
the same. This can be addressed in any number of ways. Some of the
memory usage is mitigated by the fact that we previously had lots of big
state caches and now we're keeping only keys instead.

* cleanups

* doc
2020-08-06 19:48:47 +00:00
Zahary Karadjov b902fddd19 Allow loading keystores produced by Lighthouse
The spec allows the description to be set to 'null'
2020-08-06 17:33:57 +03:00
tersec 81b3c0ea40
update spec refs to v0.12.2 (#1457)
* update spec refs to v0.12.2 and change a .len.uint64 to .lenu64

* pull back from any non-pure-comment changes, since Jenkins is being wonky
2020-08-06 13:05:13 +00:00
Zahary Karadjov 009e9a6a41 Fix 'make test' 2020-08-06 14:49:58 +03:00
protolambda e90c5440e8 make eth1 distance runtime configurable 2020-08-06 14:49:58 +03:00
Dustin Brody 8618cbdf38 keep representations consistent for tests 2020-08-04 23:00:33 +03:00
Dustin Brody e8c31e4ce8 just in case 2020-08-04 23:00:33 +03:00
Dustin Brody 5e3af9a458 fix validator cache keys 2020-08-04 23:00:33 +03:00
Dustin Brody 2c7d207a72 make test passes 2020-08-04 23:00:33 +03:00
Dustin Brody 9c65c50249 use case statement to avoid runtime doAssert 2020-08-04 23:00:33 +03:00
Dustin Brody c142de4b7f be more consistent about pubkeys fed to verify_foo_signature() not being separately initialized, while pubkeys, generally, used for matching purposes, elsewhere explicitly initialized 2020-08-04 23:00:33 +03:00
Dustin Brody db397cc5f3 lazily initialize validator public keys, which were a third or more of sync CPU usage 2020-08-04 23:00:33 +03:00
tersec a979568d74
switch mapIt/find to in-place linear search (#1436) 2020-08-04 17:15:27 +00:00
Jacek Sieka cf8cd8321b
Revert "Lazy crypto [alt design #1369] (#1371)" (#1435)
This reverts commit 023f7f4518.
2020-08-04 17:15:13 +00:00
Zahary Karadjov 4deed117a9
Hotfix: Allow 'make madella' to work with the latest Medalla metadata 2020-08-04 17:55:37 +03:00
tersec df80071bcf
update attestation and block validation to v0.12.2; clean up getAncestorAt()/get_ancestor() (#1417)
* update attestation validation to v0.12.2; clean up getAncestorAt()/get_ancestor()

* update beacon block validation to v0.12.2
2020-08-03 19:47:42 +00:00
tersec f2dd57db1e
increase allowed missing validators in verifyFinalization mode (#1414) 2020-08-03 05:31:35 +00:00
Zahary Karadjov c882b7c2f3 Add Scrypt support in the Keystores 2020-08-02 23:00:43 +03:00
Zahary Karadjov 1aba7aed6d Updated Keystore test vectors 2020-08-02 23:00:43 +03:00
Zahary Karadjov c293254ded Add 'deposits import' command; Switch to NJS when loading the keystores and improve the data validation 2020-08-02 23:00:43 +03:00
tersec 85febf072e
assert on missing active validators' attestations when verifying finalization (#1387)
* assert on missing > 10 supposed-to-be-active validators' attestations in finalization verification mode

* clean up state_sim imports
2020-08-01 18:24:25 +00:00
tersec e0a6f58abe
convert 10 v0.12.1 spec refs to v0.12.2 (#1406) 2020-07-31 09:59:14 +00:00
Viktor Kirilov c032366547
removed the BlockPool type and all of the proxy functions around it (#1401)
* removed the BlockPool type and all of the proxy functions around it - passing the chain DAG and the quarantine explicitly where appropriately - they don't need to be bundled in a type

* fixed the build after the rebase
2020-07-30 21:18:17 +02:00
tersec 254a510c0d
mark .borrow. procs as noSideEffect (#1402) 2020-07-30 14:47:59 +02:00
tersec 99f2d8e06c
update 14 v0.12.1 spec refs to v0.12.2 (#1400) 2020-07-30 09:47:57 +00:00
Mamy Ratsimbazafy 023f7f4518
Lazy crypto [alt design #1369] (#1371)
* Lazy loading of crypto objects

* Try to fix incorrect field access by hiding fields but no luck. SSZ/Chronicles/macro bug?

* Fix incorrect blsValue access. was "aggregate" not "chronicles"

* Fix tests that rely on the internal BLSValue representation
2020-07-29 18:13:05 +00:00
tersec d97cc35d30
switch 14 v0.12.1 spec refs to v0.12.2 spec refs (#1395) 2020-07-29 12:47:03 +00:00
tersec a07dab300c
verifyFinalization checks for justification being consistent with later finalization (#1386) 2020-07-28 17:35:32 +00:00
tersec b45de824a4
convert 22 v0.12.1 spec refs to v0.12.2 and remove 1 unnecessary spec ref (#1384) 2020-07-28 14:55:46 +00:00
Jacek Sieka 157ddd2ac4
Fork choice fixes 5 (#1381)
* limit attestations kept in attestation pool

With fork choice updated, the attestation pool only needs to keep track
of attestations that will eventually end up in blocks - we can thus
limit the horizon of attestations that we keep more aggressively.

To get here, we expose getEpochRef which gets metadata about a
particular epochref, and make sure to populate it when a block is added
- this ensures that state rewinds during block addition are minimized.

In addition, we'll use the target root/epoch when validating
attestations - this helps minimize the number of different states that
we need to rewind to, in general.

* remove CandidateChains.justifiedState

unused

* remove BlockPools.Head object

* avoid quadratic quarantine loop

* fix
2020-07-28 13:54:32 +00:00
Zahary Karadjov 40ea9e9886 Integrate the Wallet support with the Deposits creation; Produce Launchpad-compatible deposits 2020-07-28 07:36:25 +03:00
Zahary Karadjov fcd412f7a1 Finish the 'create wallet' command; Addresses #1319 2020-07-28 07:36:25 +03:00
tersec dad3dd5809
update 111 v0.12.1 beacon chain spec refs to v0.12.2 (#1380) 2020-07-27 18:49:46 +02:00
Jacek Sieka 48cebc7157
spec cleanups (#1379)
* add helper for beacon committee length (used for quickly validating
attestations)
* refactor some attestation checks to do cheap checks early
* validate attestation epoch before computing active validator set
* clean up documentation / comments
* fill state cache on demand
2020-07-27 16:04:44 +00:00
tersec 20a2525390
v0.12.2 beacon chain protocol update (#1378) 2020-07-27 12:59:57 +02:00
tersec 90708a8287
Prefer converting int` to uint64 and switch foo.len.uint64 to .len64 (#1375)
* avoid converting from uint64 to int, and where most feasible, int type conversion at all

* .len.uint64 -> .len64

* fix 32-bit compilation

* try keeping state_sim loop variable/bounds as int for 32-bit Azure

* len64 -> lenu64
2020-07-26 20:55:48 +02:00
Mamy Ratsimbazafy e5a76b0047
Stopgap don't print private key in shortlog (#1372) 2020-07-26 10:27:11 +02:00
Jacek Sieka e0a18a3105
cache beacon committee size calculation (#1363)
* cache beacon committee size calculation

this fixes a bug in get_validator_churn_limit as well

* fix

* make committee counts consistently uint64

mixing feels like the worst of the two worlds
2020-07-23 19:01:07 +02:00
Jacek Sieka fb2f742972
Fork choice fixes 2 (#1356)
* fork choice cleanup

* enable v2 pruning
* prefer `get_current_epoch`
* fix finalization check to use correct epoch

* small cleanups

* add `count_active_validators`
* remove misleading logs
* fix justified checkpoint slot calculation in rpc
2020-07-22 23:01:44 +02:00
tersec 6b77f3dda5
update compute_subnet_for_attestation() to use https://github.com/ethereum/eth2.0-specs/pull/1876 signature, which isn't in v0.12.1, which works with lookahead (#1346) 2020-07-22 08:04:21 +00:00
tersec 4a9a7be271
faster syncing (#1348)
* maybe faster syncing

* 80-character lines

* remove instrumentation debugEchos; fix target attestation epoch in attestation pool validation

* use the epoch-granularity matching in attestation.addResolved(...)
2020-07-22 09:51:45 +02:00
tersec 83abbcb917
drop get_attesting_indices()/get_unslashed_attesting_indices() from 15% to 1% of workload at block_sim at 100k validators (#1351) 2020-07-21 18:35:43 +02:00
Jacek Sieka 8b01284b0e
cache block hash (#1329)
hash_tree_root was turning up when running beacon_node, turns out to be
repeated hash_tree_root invocations - this pr brings them back down to
normal.

this PR caches the root of a block in the SignedBeaconBlock object -
this has the potential downside that even invalid blocks will be hashed
(as part of deserialization) - later, one could imagine delaying this
until checks have passed

there's also some cleanup of the `cat=` logs which were applied randomly
and haphazardly, and to a large degree are duplicated by other
information in the log statements - in particular, topics fulfill the
same role
2020-07-16 15:16:51 +02:00
Viktor Kirilov 4767eba465 - properly distributing the keys between BNs & VCs - fixing finalization - will switch to ON by default as soon as finalization becomes overall stable
- added a new altona-vc Makefile target which uses a separate VC process
2020-07-15 16:18:21 +03:00
tersec 26e893ffc2
restore EpochRef and flush statecaches on epoch transitions (#1312)
* restore EpochRef and flush statecaches on epoch transitions

* more targeted cache invalidation

* remove get_empty_per_epoch_cache(); implement simpler but still faster get_beacon_proposer_index()/compute_proposer_index() approach; add some abstraction layer for accessing the shuffled validator indices cache

* reduce integer type conversions

* remove most of rest of integer type conversion in compute_proposer_index()
2020-07-15 12:44:18 +02:00
tersec 4aefd8b021
fix over/underflow in proposerslashing processing (#1325) 2020-07-15 10:38:45 +00:00
zah 0be77f9cbc
Restore compilation with custom presets (#1309)
This also assigns precise types to the constants in the minimal
and mainnet presets in order to reduce the chance of compilation
errors when custom presets are used (previously, only the custom
presets have precisely assigned types for the constants).
2020-07-13 16:44:58 +02:00
Zahary Karadjov 74396747a3 Address review comments; Add a test 2020-07-12 21:01:31 +03:00
Zahary Karadjov 93b04bc214 Add an option for graffiti customization 2020-07-12 21:01:31 +03:00
tersec 853bd5b799
quick workaround for epochref cache issue (#1296)
* quick workaround for epochref cache issue

* disable assertion which doesn't work without epochref caches

* get local testnets and altona running again
2020-07-12 17:09:49 +02:00
Jacek Sieka d6f317950c
Compute state root instead of loading from database (#1297)
htr is fast now, so hitting the database to load the state root is no
longer motivated - the more simple code is less vulnerable to database
corruption.
2020-07-10 22:47:39 +02:00
Zahary Karadjov c06c60dcaf
Merge branch 'devel' into cli-reforms 2020-07-10 16:23:27 +03:00
tersec 5d5957fd7e
fix attestation handling bottlenecks in slot processing and quadratic-time inclusion delay rewards/palties in epoch processing; create infrastructure for dynamically (un)subscribing to/from attestation topics (#1294) 2020-07-10 09:24:04 +00:00
Zahary Karadjov 3ec6a02b12
Merge devel and resolve conflicts 2020-07-10 02:02:40 +03:00
tersec 61b0b5af17
update most remaining non-fork-choice spec refs, updating code where necessary (#1292)
* update most of the remaining non-fork-choice spec refs, updating code where necessary

* revert presumably harmless compute_signing_root() change, but this way, keep things really unchanged outside inspector
2020-07-09 11:43:27 +00:00
Mamy Ratsimbazafy 3cdae9f6be
Dual headed fork choice [Revolution] (#1238)
* Dual headed fork choice

* fix finalizedEpoch not moving

* reduce fork choice verbosity

* Add failing tests due to pruning

* Properly handle duplicate blocks in sync

* test_block_pool also add a test for duplicate blocks

* comments addressing review

* Fix fork choice v2, was missing integrating block proposed

* remove a spurious debug writeStackTrace

* update block_sim

* Use OrderedTable to ensure that we always load parents before children in fork choice

* Load the DAG data in fork choice at init if there is some (can sync witti)

* Cluster of quarantined blocks were not properly added to the fork choice

* Workaround async gcsafe warnings

* Update blockpoool tests

* Do the callback before clearing the quarantine

* Revert OrderedTable, implement topological sort of DAG, allow forkChoice to be initialized from arbitrary finalized heads

* Make it work with latest devel - Altona readyness

* Add a recovery mechanism when forkchoice desyncs with blockpool

* add the current problematic node to the stack

* Fix rebase indentation bug (but still producing invalid block)

* Fix cache at epoch boundaries and lateBlock addition
2020-07-09 11:29:32 +02:00
Dustin Brody 4140b3b9d9 update 29 spec refs to v0.12.1 2020-07-08 20:49:25 +00:00
Zahary Karadjov 318b225ccd
Merge devel and resolve the conflicts 2020-07-08 15:36:03 +03:00
Zahary Karadjov 5d056ba667
Switch to uint64 as the default type for preset values 2020-07-08 14:00:40 +03:00
Viktor Kirilov 1482b0430d - work towards more REST API endpoints being implemented
- testnets can now be launched with a separate validator client - make altona SCRIPT_PARAMS="--separateVC"
- reverted the ctrl+C signal handler code reuse - not necessary for the VC anyway (default is good enough)
- added a bit more logging in the VC
- removed unnecessary code in the VC - connect() just parses the address & port...
- fixed a couple more VC issues - when fetching the duties for an epoch fails on the BN side ==> the VC shouldn't be left in a broken state
- documented the currently supported json-rpc endpoints
- added more checks on the BN side for the API - bounds-checking the requests & also checking if the BN itself is synced
- other cleanup

currently a local sim doesn't finalize, but participation in the altona network with a separate VC is painless and works just as well as with in-process validators in a BN
2020-07-08 13:29:03 +03:00
Dustin Brody fc8502c54e halve memory usage from state caches 2020-07-08 10:21:41 +00:00
Dustin Brody 8ac613421d only calculate total_balance once per attestation delta application 2020-07-08 07:45:21 +00:00
Dustin Brody 3c030d945d allow 50k validators on block_sim by avoiding repeated get_total_active_balance() in get_proposer_reward() 2020-07-08 07:45:21 +00:00
Zahary Karadjov c4af4e2f35
Working test suite with run-time presets 2020-07-08 02:02:14 +03:00
Jacek Sieka 6fe0a623f5
Crypto rng (#1284)
* use bearssl rng throughout

* bump

* bump

* move keygen out of crypto
2020-07-07 17:51:02 +02:00
Zahary Karadjov 441ae9b58c
Introduce a RuntimePreset object for the overridable testnet preset values 2020-07-03 22:29:23 +03:00
tersec a92276d510
adopt Result[void, string] in place of some bool return signatures (#1275)
* adopt Result[void, string] in place of some bool return signatures

* string -> cstring to reduce memory allocations; ensure all err() strings are constants, with contextual information from higher-level callers

* logScope usage fixes

* homogenize err() reporting convention

* invalid signature in deposit isn't an error
2020-07-03 17:03:14 +00:00
tersec c64737e7f2
implement aggregated attestation receiving/validating (#1272)
* implement aggregated attestation receiving/validating

* document the conditions without explicit implementations in isValidAggregatedAttestation()
2020-07-02 16:15:27 +00:00
Jacek Sieka f3e92762e3
add tests for unviable blocks (#1271)
* add tests for unviable blocks

also enable finalization tests in all test configs - they're plenty fast
now
also fix newClone for non-rvo cases. sigh.

* fixes
2020-07-01 19:00:14 +02:00
tersec b4db2ad693
remove v0.11.3 support; add block_sim to CI (#1253)
* remove v0.11.3 support; add block_sim to CI

* rm stray PERSISTENT_COMMITTEE_PERIOD

* remove TopicFilter.InteropAttestations

* bump two comment-spec-refs to v0.12.1
2020-06-29 18:08:58 +00:00
Zahary Karadjov 8140b4458c Working local sim with WAIT_GENESIS=yes 2020-06-29 02:18:48 +03:00
Zahary Karadjov 31e31bb30c Switch to monitoring of Blocks; More accurate genesis detection and faster Eth1 syncing 2020-06-29 02:18:48 +03:00
zah 578ea2013a
implement spev v0.12.1 in initialize_beacon_state_from_eth1 (#1236) 2020-06-25 16:24:55 +02:00
Jacek Sieka 1301600341
Trusted blocks (#1227)
* cleanups

* fix ncli state root check flag
* add block dump to ncli_db
* limit ncli_db benchmark length
* tone down finalization logs

* introduce trusted blocks

We only store blocks whose signature we've verified in the database - as
such, there's no need to check it again, and most importantly, no need
to deserialize the signature when loading from database.

50x startup time improvement, 200x block load time improvement.

* fix rewinding when deposits have invalid signature
* speed up ancestor iteration by avoiding copy
* avoid deserializing signatures for trusted data
* load blocks lazily when rewinding (less memory used)

* chronicles workarounds

* document trustedbeaconblock
2020-06-25 12:23:10 +02:00
Mamy Ratsimbazafy 902093f57c
Revert "Dual headed fork choice [Reloaded] (#1223)" (#1234)
This reverts commit 6836d41ebd.
2020-06-25 11:36:03 +02:00
Zahary Karadjov 384e512031 Implement EIP-2386 wallets; Progress towards a CLI for interactive wallet creation
For more information:
4494da0966/EIPS/eip-2386.md (specification)
2020-06-24 23:06:05 +03:00
Mamy Ratsimbazafy 6836d41ebd
Dual headed fork choice [Reloaded] (#1223)
* Dual headed fork choice

* fix finalizedEpoch not moving

* reduce fork choice verbosity

* Add failing tests due to pruning

* Properly handle duplicate blocks in sync

* test_block_pool also add a test for duplicate blocks

* comments addressing review
2020-06-24 20:24:36 +02:00
tersec 1b3c19dc21
default to v0.12.1 beacon chain spec, keeping witti working (#1222)
* default to v0.12.1 beacon chain spec, keeping witti working

* a couple of fixes that help for the Onyx testnet
2020-06-23 19:40:44 +00:00
tersec 807b920c19
state_transition implements the spec fairly directly (#1220) 2020-06-23 13:54:24 +00:00
Zahary Karadjov e9d68e2f7b Add simulation mode that bootstraps from an Eth1 ganache instance 2020-06-22 17:30:04 +03:00
Dustin Brody 62a7e7fede fix is_valid_indexed_attestation() to check for too-high attestation indices 2020-06-22 14:15:51 +00:00
tersec 7b4e129316
re-add minimal constant checking; organize presets by spec version (#1203)
* re-add minimal preset constant checking; organize presets to better support multiple spec versions

* bump spec ref

* increase Azure timeout to 90 minutes to accomodate Nim compiler building
2020-06-20 09:12:45 +02:00
Dustin Brody 4ecbc655ea switch 11 beaconstate.nim spec refs from v0.11.x to v0.12.1 2020-06-19 11:54:37 +00:00
Viktor Kirilov 72dfe7f578 - updated the validator shell script after the keystore changes
- better logging & retrying requests on the VC side if the BN fails for some reason
- VC now fetches the attestation duties 1 epoch in advance - in the future it will tell the BN to subscribe to the appropriate attestation topics in advance based on that info
- a bunch of other code cleanup & fixes such as better naming for consoles when using multitail, etc.

reviewed in PR #1184 - proper review of the API & VC are pending
2020-06-19 12:21:22 +03:00
tersec dc1a565b3f
support v0.12.1 attestation topics in beacon node/inspector subscribing (#1187)
* support v0.12.1 attestation topics in beacon node and inspector subscribing

* bump is_valid_merkle_branch() spec ref
2020-06-18 15:10:25 +02:00
Dustin Brody db870fead4 update 20 beacon chain protocol spec refs 2020-06-18 08:39:28 +00:00
tersec ee9f4a2e3f
remove skipMerkleValidation and skipBlockParentRootValidation (#1197) 2020-06-18 07:56:47 +02:00
Dustin Brody ffca27b45f update 24 v0.11.x spec refs to v0.12.1 2020-06-17 12:11:03 +00:00
Dustin Brody ec85a8bdbf update a dozen unchanged spec refs to v0.12.1 2020-06-17 09:25:54 +00:00
Jacek Sieka 5c25d23ef1 eth2hash -> eth2digest
hash in nim is the insecure hash-map helper - might as well use `digest`
consistently to mark the difference
2020-06-16 13:30:21 +00:00
Jacek Sieka 89e4819ce9
collect signature production and verificaiton in one place (#1179)
* collect signature production and verificaiton in one place

Signatures are made over data and domain - here we collect all such
activities in one place.

Also:
* security: fix cast-before-range-check
* log block/attestation verification consistently
* run block verification based on `getProposer` in its own history
* clean up some unused stuff

* import

* missing raises
2020-06-16 07:45:04 +02:00
Dustin Brody 8a2c796e83 switch 12 unchanged spec references from 0.11.x to 0.12.1; line-wrap and remove pointless-to-counterproductive "return" 2020-06-15 10:10:31 +00:00
Jacek Sieka 360ebd705f db: compress with snappy
* 3gb vs 12gb for 4000 epochs of witti
* 3-4x sync blocks/sec performance improvement on my FDE SSD drive
* generate less quirky code for primitive types
2020-06-14 11:33:00 +03:00
Dustin Brody 1d1e90adf4 Update 36 spec refs from 0.11.x to 0.12.1 2020-06-12 19:41:03 +00:00
Jacek Sieka 78b767f645
avoid genericAssign for beacon node types (#1166)
* avoid genericAssign for beacon node types

ok, I got fed up of this function messing up cpu measurements - it's so
ridiculously slow, it's sad.

before, while syncing:

```
40,65%  beacon_node_shared_witti_0  [.]
genericAssignAux__U5DxFPRpHCCZDKWQzM9adaw
   9,02%  libc-2.31.so                [.] __memmove_avx_unaligned_erms
   7,07%  beacon_node_shared_witti_0  [.] BIG_384_58_monty
   5,19%  beacon_node_shared_witti_0  [.] BIG_384_58_mul
   2,72%  beacon_node_shared_witti_0  [.] memcpy@plt
   1,18%  [kernel]                    [k] rb_next
   1,17%  beacon_node_shared_witti_0  [.] genericReset
   1,06%  [kernel]                    [k] map_private_extent_buffer
```

after:

```
  24,88%  beacon_node_shared_witti_0  [.] BIG_384_58_monty
  20,29%  beacon_node_shared_witti_0  [.] BIG_384_58_mul
   3,15%  beacon_node_shared_witti_0  [.] BIG_384_58_norm
   2,93%  beacon_node_shared_witti_0  [.] BIG_384_58_add
   2,55%  beacon_node_shared_witti_0  [.] BIG_384_58_sqr
   1,64%  beacon_node_shared_witti_0  [.] BIG_384_58_mod
1,63%  beacon_node_shared_witti_0  [.]
sha256Transform__BJNBQtWr9bJwzqbyfKXd38Q
   1,48%  beacon_node_shared_witti_0  [.] FP_BLS381_add
   1,39%  beacon_node_shared_witti_0  [.] BIG_384_58_sub
   1,33%  beacon_node_shared_witti_0  [.] BIG_384_58_dnorm
   1,14%  beacon_node_shared_witti_0  [.] FP2_BLS381_mul
   1,05%  beacon_node_shared_witti_0  [.] BIG_384_58_cmove
1,05%  beacon_node_shared_witti_0  [.]
get_shuffled_seq__4uncAHNsSG3Pndo5H11U9aQ
```

* better field iteration
2020-06-12 21:10:22 +02:00
Jacek Sieka 42832cefa8
Small fixes (#1165)
* random fixes

* create dump dir on startup
* don't crash on failure to write dump
* fix a few `uint64` instances being used when indexing arrays - this
should be a compile error but isn't due to compiler bugs
* fix standalone test_block_pool compilation
* add signed block processing in ncli

* reuse cache entry instead of allocating a new one

* allow for small clock disparities when validating blocks
2020-06-12 18:43:20 +02:00
Zahary Karadjov fdaf419e41 Address review comments 2020-06-11 17:40:08 +03:00
Zahary Karadjov a8113cf2bc Restore the local sim to a working state 2020-06-11 17:40:08 +03:00
Zahary Karadjov 17343442ea Implement more of the KeyStore spec and integrate it in the beacon node 2020-06-11 17:40:08 +03:00
Viktor Kirilov e7febc2e2b can finalize successfully with external VCs! + some other cleanup 2020-06-10 13:50:50 +03:00
Viktor Kirilov 3bae40ae91 - extracted the commands to run a VC into a separate run_validator.sh script
- each BN gets half of its previous validators as inProcess and the other half goes to the respective VC for that BN - using separate data dirs where the keys are copied
    - also removed a few command line options which are no longer necessary
- block proposals originating from a VC are propagated from one BN to the rest properly
- other cleanup & moving code back to  since it is no longer used elsewhere
2020-06-10 13:50:50 +03:00
Mamy Ratsimbazafy ce897fe83f
[Split fork choice PR] Derisk-ed attestation checks changes (#1154)
* Derisked attestation pool improvements

* tune down frequent logs

* VoteTracker logging
2020-06-10 08:58:12 +02:00
Dustin Brody a9f73ac8fb poke 2020-06-09 18:39:05 +00:00
Dustin Brody 8f7541959a allow compile-time switching between v0.11.3 and v0.12.1 2020-06-09 18:39:05 +00:00
Dustin Brody 36f15a3688 complete re-implmentation of reasonable asymptotic-complexity algorithms from 0.11.3 2020-06-09 18:39:05 +00:00
Dustin Brody 35f67cec60 remove the scaffolding get_empty_per_epoch_cache() calls and the pointless construction and return/discarding of zero-filled arrays 2020-06-09 18:39:05 +00:00
Dustin Brody cc13153fc6 initial 0.12.1 beacon chain spec 2020-06-09 18:39:05 +00:00
Zahary Karadjov 32bc4d9096 Use {.push raises.} in spec/datatypes
This was an attempt to trigger a previously reported crash, but
everything turned out to be fine.
2020-06-05 19:33:30 +03:00
Dustin Brody 3cb7896bab 12x speedup on state sim with 100k validators sans BLS by caching get_beacon_proposer_index(...) 2020-06-04 17:07:51 +00:00
Jacek Sieka 56ffb696be
reorder ssz (#1099)
* reorder ssz

* split into hash_trees and ssz_serialization, roughly, for hashing and
IO
* move bitseqs into ssz (from stew)
* clean up imports

* docs, imports
2020-06-03 15:52:02 +02:00
Jacek Sieka 07393c8de6
Ssz cache (#1084)
* wip: cache

* cache lists and arrays of complex objects (5x block processing speed
on ncli_db)

trivial baseline cache that stores tree in flat memory structure

* support array of uint64

* work around type issues

* more type compiler bug workarounds

* cache balances, more type fixes

* index type

* ncli_db: add validation flag, better ux

* int64 fixes

* test fix

* "oops"

```
647.913,        0.000,      647.913,      647.913,            1,
Initialize DB
0.540,        0.402,        0.340,        9.451,          619,
Load block from database
40.268,        0.000,       40.268,       40.268,            1,
Load state from database
0.498,        0.150,        0.343,        0.930,          596,
Apply block
3.548,       11.005,        0.729,       54.022,           23,
Apply epoch block
```

* support all basic types

* cleanups

* a few more cleanups
2020-06-03 07:57:17 +02:00
tersec b7bb62d48b
use shuffled validator cache in more places; cleanups (#1095) 2020-06-03 07:42:08 +02:00
Jacek Sieka 1d9a58f1e8
a few more cleanups 2020-06-02 22:10:45 +02:00
Dustin Brody 1a6b83a15f cleanups 2020-06-01 19:45:21 +00:00
Zahary Karadjov ef90a45509
Bump testutils 2020-06-01 16:58:28 +03:00
Jacek Sieka fec01829c4
cache balances, more type fixes 2020-06-01 14:52:52 +02:00
Jacek Sieka be92842944
more type compiler bug workarounds 2020-06-01 14:52:52 +02:00
Jacek Sieka 4fc9a84c57
support array of uint64 2020-06-01 14:52:52 +02:00
Jacek Sieka f1dcee2749
wip: cache
* cache lists and arrays of complex objects (5x block processing speed
on ncli_db)

trivial baseline cache that stores tree in flat memory structure
2020-06-01 14:52:50 +02:00
tersec a327e8581b
switch state transition caching to match EpochRef (#1089)
* switch state transition caching usage to shuffled active validator indices to match EpochRef

* refactor the EpochRef -> StateCache transformation; elide pointless mapIt

* limit state passed between get_beacon_committee(...) and compute_committee(...)

* tweaks
2020-06-01 09:44:50 +02:00
Zed c570abf602 Check iv length for decryption 2020-05-29 16:38:55 +03:00
Zed 80ce960afa Minor improvements 2020-05-29 16:38:55 +03:00
Zed 49c083d5b7 Catch all exceptions 2020-05-29 16:38:55 +03:00
Zed 14ad100b45 Address comments 2020-05-29 16:38:55 +03:00
Zed 8496e20a78 Implement EIP 2335 compliant keystore
Closes #1024
2020-05-29 16:38:55 +03:00
tersec b5f45db5e9
keep cache of per-epoch items in block pool (#1068)
* plumbing between block pool and state transition functions around active validator indices and committees

* have shared epochrefs followed by blockref tree while allowing for skipped slots

* factor out the epoch info extraction; document how the EpochRef follows forks
2020-05-29 08:10:20 +02:00
Jacek Sieka f53b55cbe0 SSZ cleanup
* be stricter about SSZ length prefix
* compute zeroHash list at compile time
* remove SSZ schema stuff
* move SSZ navigation to ncli
* cleanup a few leftover openArray uses
2020-05-28 19:15:16 +03:00
Viktor Kirilov cefd525ab3 more work on the BN/VC split
- fixed comments from the last review
- getting more data VIA RPC, moved some code back into the BN only
- attestation duties being requested as well
2020-05-28 13:01:42 +03:00
Dustin Brody 0929d90d93 unexport candidate_chains.init; some spec version bumps 2020-05-26 05:06:37 +00:00
Viktor Kirilov 8760494c72 first batch of work towards the VC/BN split:
- we have a new binary which connects via RPC to the respective BN and has an internal clock - waking it up on every slot
- the BN has a new option called --external-validators and currently in order to have the VC binaries to run we need to pass EXTERNAL_VALIDATORS=yes to make
- factored some code out of beacon_node.nim for easier reuse in validator_api.nim and validator_client.nim
- the VC loads its associated private keys from the datadir for its BN
- most of the validator API calls have been implemented as a stub.
- the VC polls its BN at the start of each epoch - getting a list of all active validators for the current epoch - and then continues to request blocks and sign them with its appropriate validators when necessary
2020-05-25 16:23:15 +03:00
Zahary Karadjov a8003e7e38 More complete integration of the List type; Detect more invalid inputs; Enable more tests 2020-05-24 19:00:34 +03:00
Dustin Brody 74e35c464f halt processing if one of the operation functions fails (failed deposits due to BLS signature issues return as not-failed, so it remains consistant with that 2020-05-24 08:20:13 +00:00
Dustin Brody 67ba11cc1a use template to more cleanly implement for_ops 2020-05-24 08:20:13 +00:00
Dustin Brody 95ee0218db switch too-strict/against-spec operations processing to use spec-based process_operations 2020-05-24 08:20:13 +00:00
Jacek Sieka f06df1cea6 remove some copies
* in makeBeaconBlock - use rollback instead
* in tests - this helps state_sim give more accurate data and makes it
30% faster
* fix some usages of raw BeaconState
2020-05-22 17:15:35 +00:00
tersec 80b8416305
skip skipped constant sanity checks also from markdown reports' perspectives (#1053)
* update more spec refs in beacon_chain/spec/presets; skip skipped constant sanity checks also from markdown reports' perspectives

* mark skipped as skipped in markdown
2020-05-21 19:56:09 +02:00
Dustin Brody b9817867f5 audit & update spec references for some slashing and rewards and penalty functions; check and update all remaining v0.11.0 spec references to v0.11.3 [skip ci] 2020-05-20 14:57:08 +00:00
Dustin Brody 3fa2f70853 update all remaining beacon chain spec references from v0.11.1 and v0.11.2 to v0.11.3 2020-05-20 08:48:18 +00:00
Dustin Brody 6c941b1ff7 spec v0.11.3 update 2020-05-19 17:16:55 +00:00
Zahary Karadjov a99977b772 Progress towards #991; Enable the distinct ssz.List type 2020-05-19 15:53:03 +03:00
tersec 8432932c11
update to spec v0.11.2 to keep compatible with Schlesi (#1036)
* update to spec v0.11.2 to keep compatible with Schlesi

* update spec/state_transition_epoch spec references to v0.11.2

* bump other spec refs
2020-05-18 23:05:27 +00:00
Dustin Brody 2ddc507e5b update datatypes v0.11.1 spec references to v0.11.2 2020-05-14 16:33:11 +00:00
Dustin Brody 57519bebac remove some unused imports, add tests for pubsub topics, and subscribe to interop attestations 2020-05-14 16:41:46 +03:00
Zahary Karadjov a739d7e8d6 Adapt SSZ to the latest FastStreams API 2020-05-13 12:18:42 +03:00
Jacek Sieka fb2e0ddbec
sync fixes (#1005)
* sync fixes

* fix Status message finalized info
* work around sync starting before initial status exchange
* don't fail block on deposit signature check failure (fixes #989)
* print ForkDigest and Version nicely
* dump incoming blocks
* fix crash when libp2p peer connection is closed
* update chunk size to 16 to work around missing blocks when syncing

* bump libp2p

* bump libp2p

* better deposit skip message
2020-05-11 18:08:52 +00:00
Ștefan Talpalaru a7a50824a1
more metrics (#1004) 2020-05-11 06:25:49 +00:00
tersec 093d298b2b
Increase finalization and finalization checking robustness (#990)
* fix some warnings related to beacon_node splitting; reimplement finalization verification more robustly; improve attestation pool block selection logic

* re-add missing import

* whitelist allowed state transition flags and make rollback/restore naming more consistent

* restore usage of update flags passed into skipAndUpdateState(...) in addition to the potential verifyFinalization flag

* switch rest of rollback -> restore
2020-05-09 12:43:15 +00:00
Ștefan Talpalaru c572f61129
bump vendor/nim-metrics 2020-05-09 01:13:57 +02:00
tersec c548b90a0e
default preset update; rm process_slots(BeaconState); some BLS skipping disabled (#963)
* re-enable BLS validation for block sanity tests

* update default preset to mainnet; remove process_slots(state: var BeaconState)

* bump spec ref to v0.11.1

* bump all v0.11.1 spec references except the actually changed is_valid_indexed_attestation(...) in spec/beaconstate to v0.11.2

* remove stack-risky/low-ROI doAssert
2020-05-04 21:07:18 +00:00
Jacek Sieka 2449d4b479
cache empty slot state root (#961)
When replaying state transitions, for the slots that have a block, the
state root is taken from the block. For slots that lack a block, it's
currently calculated using hash_tree_root which is expensive.

Caching the empty slot state roots helps us avoid recalculating this
hash, meaning that for replay, hashes are never calculated. This turns
blocks into fairly lightweight "state-diffs"!

* avoid re-saving state when replaying blocks
* advance empty slots slot-by-slot and save root
* fix sim randomness
* fix sim genesis filename
* introduce `isEpoch` to check if a slot is an epoch slot
2020-05-03 19:44:04 +02:00
tersec cd939d4606
fix spec/beaconstate compile error in the presence of other PRs' SSZ tightening (#952) 2020-04-30 09:01:55 +00:00
tersec 3a56ddc5c4
begin removing plain BeaconState versions of state transition functions (#951)
* remove near-duplicate code paths: process_slot(), process_slots(), and state_transition() for BeaconState are now wrappers around the HashedBeaconState versions

* convert tests/test_state_transition.nim to use HashedBeaconState

* convert mocking infrastructure and spec_block/epoch_processing tests to use HashedBeaconBlock, and remove thus unused process_slot*(state: var BeaconState)
2020-04-30 08:44:19 +02:00
Jacek Sieka c74ba5c0c6
ssz: move ref support outside (#943)
* ssz: move ref support outside

Instead of allocating ref's inside SSZ, move it to separate helper:

* makes `ref` allocations explicit
* less magic inside SSZ
* `ref` in nim generally means reference whereas SSZ was loading as
value - if a type indeed used references it would get copies instead of
references to a single value on roundtrip which is unexpected

TODO: EF tests would benefit from some refactoring since they all do the
same thing practically..

Co-authored-by: Zahary Karadjov <zahary@gmail.com>
2020-04-29 22:12:07 +02:00
Jacek Sieka 28d6cd2524
avoid memory allocations and copies when loading states (#942)
* rolls back some of the ref changes
* adds utility to calculate stack sizes
* works around bugs in nim exception handling and rvo
2020-04-28 10:08:32 +02:00
Zahary Karadjov 80b538452e Updated the custom preset loader to the latest spec; Working makefile target for connecting to the Schlesi testnet 2020-04-27 23:22:54 +03:00
tersec be475a82d7
catch invalid 0-raw-byte Bitlist SSZ inputs during parsing (#936)
* catch invalid 0-raw-byte Bitlist SSZ inputs during parsing

* bump a couple more spec refs to v0.11.1
2020-04-26 16:26:53 +00:00
Zahary Karadjov 4e9fa51ae9 Introduce BeaconNodeRef and use it in all the right places 2020-04-26 13:04:53 +03:00
Zahary Karadjov fdcbfdff05 Pass the test suite with a BeaconState ref type 2020-04-26 13:04:53 +03:00
Zahary Karadjov 740b76d152 Spec-compliant implementation of Eth1 monitoring; Eth1-enabled local sim
BEWARE! This commit will trigger a stack overflow during local sim
2020-04-26 13:04:53 +03:00
tersec 65353bab76
add another check for inconsistent aggregation and committee length (#927)
* add another check for inconsistent aggregation and committee length, since ncli_transition bypasses process_attestation(...)/check_attestation(...) and calls almost directly into process_epoch(...)

* bump validator functions to v0.11.1 spec references

* bump some spec references to v0.11.1

* poke
2020-04-24 09:16:40 +02:00
Viktor Kirilov 0079242457 again exported the right overloads for serializing hashes properly to json (and not as byte arrays) and updating nim-eth 2020-04-23 20:46:13 +03:00
Viktor Kirilov 4c66999a43 removing the json serialization change - fails to compile with -d:testnet_servers_image and LOG_LEVEL=TRACE and isn't worth the time at the moment 2020-04-23 20:46:13 +03:00
Viktor Kirilov 1575ded589 exported the right overloads for serializing hashes properly to json (and not as byte arrays) 2020-04-23 20:46:13 +03:00
tersec 7cd4b0bfae
reduce stack space usage in process_final_updates(...) (#925)
* reduce stack space usage in process_final_updates(...) to avoid fuzzed segfault in https://github.com/status-im/nim-beacon-chain/issues/921

* document motivation behind manually constructing hash_tree_root of a HistoricalBatch
2020-04-23 13:39:38 +02:00
Jacek Sieka ed74770451
spec: regulate exceptions (#913)
* spec: regulate exceptions

* a few more simple raises
2020-04-22 07:53:02 +02:00
tersec 8a72ae89b9
fix mainnet finalization (#906)
* fix mainnet finalization and swith eth2_network_simulation to a kind of small-mainnet profile

* Fix slot reference in trace logging

* bump a couple of spec refs from v0.11.0 to v0.11.1

* bump another spec ref to v0.11.1, one more try at Jenkins test vector download CI issue

* fix other slot reference in trace logging and skip past single-block/multi-slot gaps to re-approach from ancestry side by state_transitioning, by requiring exact match on both root hash and slot for fast path

* make more precise the fast path condition

* redo logic to make uniform with BeaconChainDB; fix chronos deprecation warning

* revert not-working replacement of deprecated chronos futures `or`

* switch testnet1 to mainnet
2020-04-20 19:27:52 +02:00
tersec 480c11f8aa
fix remaining block sanity test vectors (#902)
* fix remaining block sanity test vectors

* bump epoch processing spec refs to v0.11.1
2020-04-18 00:09:34 +02:00
Dustin Brody 164d171bd9
revert previous commit 2020-04-17 20:27:42 +02:00
Dustin Brody ef92529feb
fix remaining block sanity test vectors 2020-04-17 20:25:18 +02:00
Dustin Brody 3df9a22ff7 fix remaining attester slashing tests from EF phase0 spec tests by checking for uniqueness 2020-04-17 16:01:32 +00:00
Dustin Brody 05d1d1246a bump a couple more eth2 phase0 spec refs to v0.11.1 2020-04-17 16:37:52 +03:00
Dustin Brody 943a7da610 bump some eth2 phase0 spec refs to v0.11.1 2020-04-17 16:37:52 +03:00
Zahary Karadjov 99f9bc4eee Fix the inspector build 2020-04-16 01:01:18 +03:00
Zahary Karadjov 672f690bf6 Implement most of the v0.11.1 networking changes 2020-04-16 01:01:18 +03:00
Dustin Brody d559f4ee89 create distinct CommitteeIndex type 2020-04-15 09:59:49 +00:00
Dustin Brody f49dbf68e6 refactor some Merkle proof generation code outside tests; remove unused variables/functions 2020-04-15 09:59:49 +00:00
tersec d25d674502
Remove more warnings, both deprecations and unused imports (#884)
* fix warnings by switching from deprecated chronos API addTimer(...) to setTimer(...) and removing especially some unnecessary chronicles and extras imports from test suite modules

* update a couple v0.10.1 spec references to v0.11.1
2020-04-11 19:41:50 +02:00
Jacek Sieka afa08c8e3c
crypto: cleanup (#882)
* crypto: cleanup

* fix several Defect-on-user-input
* make crypto interface more similar to secp
* use `crypto.nim` in all of nbc

* digest: raises

* fix

* vendor
2020-04-11 10:51:07 +02:00
tersec 31bf8fa408
remove all but one UnusedImport warning (#881)
* remove all but one UnusedImport warning

* bump a few more spec version references from v0.10.1 to v0.11.1

* more v0.10.1 spec reference updates/removals

* yet more v0.10.1 spec reference updates
2020-04-10 16:06:24 +02:00
tersec ccbbce79a9
Mostly remove skipMerkleValidation by fixing Merkle proof construction/usage (#879)
* refactor and fix merkle proof construction in test suite and thereby remove most remaining skipMerkleValidation flags, now unnecessary

* a few non-semantic comment update/removals
2020-04-10 15:59:17 +02:00
Jacek Sieka 0f47c76b50
reenable block bls verification (#875)
* finalizing state_transition

* cleanup

Co-authored-by: Dustin Brody <tersec@users.noreply.github.com>
2020-04-09 09:41:02 +00:00
Dustin Brody 2771deadfc re-add test_interop to all_tests and mark several v0.10.1 phase 0 spec references as v0.11.1 2020-04-07 13:16:55 +03:00
Dustin Brody e7de8aae20 fix/restore 0.11.1 BeaconState genesis interop: (a) use GENESIS_FORK_VERSION properly in BeaconState init; (b) use GENESIS_FORK_VERSION to be compatible with zcli/zrnt initial eth1 deposits; (c) let List[foo] work with 64-bit ints, so BeaconState.validators SSZ-serializes properly; and (d) update test_interop to use new spec/crypto API 2020-04-07 13:16:55 +03:00
Dustin Brody 60df05a420 mark several v0.10.1 spec references as v0.11.1-compatible 2020-04-04 13:43:04 +00:00
tersec ea59f2ecf2
comments-only changes: update a bunch of v0.10.1 spec references to v0.11.0 and explain rationale for extended validation design in code (#847) 2020-04-01 11:41:39 +00:00
tersec 6eb4f1f39d
initial attestation aggregation (#769)
* initial attestation aggregation

* fix usage of committee index, vs index in committee; uniformly set trailing/following distance; document how the only-broadcast-if mechanism works better and what aggregation already happens, not otherwise sufficiently clear; use correct BlockSlot across epoch boundaries

* address inconsistent notion of which slot in past to target for aggregate broadcast; follow 0.11.x aggregate broadcast p2p interface topic

* Fix get_slot_signature(...) call after get_domain(...) change required genesis_validators_root

* mark all spec references which aren't dealt with in other PRs as v0.11.1

* update two more spec refs to v0.11.1
2020-04-01 09:59:55 +00:00
tersec cd388bc9bb
extended validation (#812)
* initial extended validation setup

* flesh out all TODO items for attestation and beaconblock verification

* fix finalization and add chronicles debugging messages

* directly use blockPool.headState rather than pointlessly updating it and document this constraint

* fix logic relating to first-attestation checking; support validating blocks across multiple forks
2020-03-31 18:39:02 +00:00
tersec daabb1b5b2
complete all (non-semantic, comment-only) 0.11.0 -> 0.11.1 beacon chain spec updates, mark all beacon chain v0.11.0 spec references as v0.11.1, and remove now unnecessary/unused UpdateFlags vars from 4 test fixtures (#841) 2020-03-30 23:40:24 +00:00
tersec f5f939bd31
0.11.1 beacon chain spec update (#836)
* initial 0.11.1 spec commit; no test regressions and finalizes in eth2_network_simulation

* with BLS 0.10/0.11 available, stop skipping attester slashing, proposer slashing, and voluntary exist operations fixture tests

* switch param orders to group state.{fork, genesis_validators_root}; bump spec/datatypes spec version for network purposes

* mark attestation construction and broadcast and some minimal/mainnet constants as 0.11.1-compatible; remove phase 1 sharding constants from minimal which don't exist in that preset
2020-03-30 11:31:44 +00:00
Zahary Karadjov 5a2a52869e Add a RPC proc similar to /spec/eth2_config 2020-03-22 19:00:09 +02:00
Zahary Karadjov adcec61081 Initial implementation of a JSON-RPC service 2020-03-22 19:00:09 +02:00
tersec 53e4b0a26c
complete (except for get_domain(...)) 0.11.0 beacon chain spec update (#822)
* complete (except for get_domain(...)) 0.11.0 beacon chain spec update

* mark compute_start_slot_at_epoch(...), is_active_validator(...), compute_signing_root(...), and get_seed(...) as 0.11.0
2020-03-22 16:03:07 +00:00
Jacek Sieka 689bcf71c4 clean up block creation
* consistently use state at new block slot to produce block
* factor out signature funcs
* fix missing block application test
2020-03-22 11:15:07 +00:00
Mamy Ratsimbazafy ccace9a034
Fix shortlog crypto (#818)
* fix shortlogs of crypto types

* Valid signature without a "real: " prefix tag

* remove 0x prefix
2020-03-19 17:18:48 +01:00
Dustin Brody f7ecd09e95 mark aggregate(...), AttesterSlashing, IndexedAttestation, AttestationData, Deposit, DepositData, VoluntaryExit, BeaconBlock, BeaconBlockHeader, BeaconState, HistoricalBatch, Fork, SigningRoot, SignedVoluntaryExit, SignedBeaconBlock, SignedBeaconBlockHeader, get_current_epoch(...), and get_randao_mix(...) as 0.11.0 2020-03-17 12:06:56 +02:00
Dustin Brody cae1c0fd1a address beacon_node TODO regarding missing proposer index; mark initiate_validator_exit(...), is_eligible_for_activation_queue(...), Attestation, DepositMessage, BeaconBlockBody, PendingAttestation, Eth1Data, and integer_squareroot(...) as 0.11.0 2020-03-17 12:06:56 +02:00
Dustin Brody ebcde664e5 mark decrease_balance(...), get_validator_churn_limit(...), network spec version, Checkpoint, compute_epoch_at_slot(...), get_active_validator_indices(...), and get_committee_count_at_slot(...) as 0.11.0 2020-03-17 12:06:56 +02:00
Dustin Brody 313109c743 align process_block(...) name with spec; fix beacon chain proposer index generation when some validators aren't active to fix the two regressing 0.11.0 block sanity tests 2020-03-17 12:06:56 +02:00
Dustin Brody 6068dd960b initial 0.11.0 spec version update 2020-03-17 12:06:56 +02:00
Mamy André-Ratsimbazafy 2aae54a15f Fix serialization of deposits 2020-03-13 12:20:41 +00:00
Mamy André-Ratsimbazafy c1c7435808 Skip tests:
- EF - attester slashings - 2 tests
- EF - sanity blocks - 1 test
- NBC - attestation - 1 test
2020-03-13 12:20:41 +00:00
Mamy André-Ratsimbazafy 28dc8a6a29 Fix mock signatures/validator keys and stack smashing 2020-03-13 12:20:41 +00:00
Mamy André-Ratsimbazafy 14bc9e60ca "init" should handle fake/invalid pubkeys and signatures 2020-03-13 12:20:41 +00:00
Mamy André-Ratsimbazafy 3c140d0c3d Try to solve the stack smashing / discriminant changes object branch 2020-03-13 12:20:41 +00:00
Mamy André-Ratsimbazafy 31bd13a5d3 init for mainchain_monitor 2020-03-13 12:20:41 +00:00
Mamy André-Ratsimbazafy bfd4df912e Update to BLS v0.10.1 2020-03-13 12:20:41 +00:00
tersec a327c82820 Revert "finalizing state_transition (#792)"
This reverts commit 91d75bdfdd.
2020-03-10 16:44:42 +00:00
Joao Gabriel Carvalho 91d75bdfdd
finalizing state_transition (#792) 2020-03-10 12:19:00 +01:00
Jacek Sieka 7902d070cd prefer endians2 2020-03-05 18:13:28 +02:00
Jacek Sieka 9a3db7a81f
[WIP] Fake bls at runtime (#735)
* Initial implementation of runtime bls skipping.

Add libnfuzz skipBLSValidation handling, check that it propagates.

* Rename skipBLSValidation -> skipBlsValidation, start skipStateRootValidation
* Replace skipValidation flags with more granular flags.

Also added skipBlockParentRootValidation flag
Mainly replaced with skipBlsValidation but also StateRoot or
BlockParentRootValidation flags where appropriate.

* Adjust interop test to pass when skipping merkle validation.
* Stop skipping validation for mainchain_monitor.
* Remove comment.
* Also skipMerkleValidation for test_beacon_chain_db.
2020-03-05 13:52:10 +01:00
Dustin Brody 3d99c7b28e mark some spec functions .used. to avoid warnings; misc cleanups 2020-02-25 09:17:06 +00:00
Dustin Brody 425b0e13da remove potential error condition in getAttestationTopic; subscribe to all topics in parallel; add test cases for larger-than-ATTESTATION_SUBNET_COUNT committee indexes 2020-02-21 19:44:03 +00:00
Dustin Brody 62584b633e update relevant portions of attestation aggregation spec version; tweak inspector 2020-02-21 19:44:03 +00:00
Dustin Brody a8567cafb5 mark out where/how AttestationData and Attestation construction are honest validator-compliant 2020-02-21 19:44:03 +00:00
Dustin Brody 32872c7ee6 bump spec version to 0.10.1 (more accurate than 0.10.0); fix inspector building 2020-02-21 19:44:03 +00:00
Dustin Brody 6c87af9aab split attestation beacon topic per honest validator spec 2020-02-21 19:44:03 +00:00
Dustin Brody bced5ca656 implement is_proposer(...) 2020-02-21 19:44:03 +00:00
Dustin Brody 4782ae06e6 implement get_committee_assignment(...) by spec transcription 2020-02-21 19:44:03 +00:00
Jacek Sieka 521b0ed6ba
Clean 20200205 (#729)
* beacon node code cleanup
* rudimentary error checking on mainnet monitor
* start client even when sending deposit
* work around missing block number exception
* connect to testnet with web3 url
* pretty-print digests in json
2020-02-07 08:13:38 +01:00
Dustin Brody 1ffc2df23d add a couple new deposit tests; fix the false-positive BLS verifications while keeping all but two tests working, despite mismatched 0.9/0.10 BLS standards; better-factor the skipping of BLS validation and Merkle tree validation 2020-02-04 18:39:38 +00:00
Dustin Brody e4922cb177 implement compute_signing_root() and mark some functions as 0.10.1-compatible 2020-02-04 12:24:30 +00:00
Dustin Brody 45dd12cf3f update process_deposit() to actually check is_valid_merkle_branch() unless skipValidation specified 2020-01-30 09:31:08 +00:00
Dustin Brody 60b1775879 update check_attestation/process_attestation() to 0.10.1 and fill in missing check 2020-01-28 12:07:33 +02:00
Dustin Brody 3efbe36c30 additional 0.10.1 updates where already-implemented code is already done 2020-01-27 12:50:54 +00:00
Dustin Brody dab107fa1e more 0.10.1 updates 2020-01-27 12:50:54 +00:00
Mamy Ratsimbazafy 1ba2cebd85 v0.10.1 (test vectors + unchanged parts) (#696)
* update to 0.10.1

* SSZ Generic and nbench uses the v0.10.1 fixtures

* Tests + spec links: v0.10.0 -> v0.10.1

* Add v0.10.1 TODO in get_latest_attesting_balance (forkchoice)

* SSZ Bytes are now ByteList

* Remove nim-result submodules that was leftover/added by mistake in the branch
2020-01-27 10:56:32 +00:00
Jacek Sieka 270c25c8b8
drop rlpx support (#679) 2020-01-24 09:32:52 +01:00
Jacek Sieka 7dc2434e0d
clean up attestation logging 2020-01-23 18:48:26 +01:00
Dustin Brody 50f633209c mark a couple dozen 0.9.2/0.9.4 spec locations as still consistent with 0.10.0; no functional changes 2020-01-22 13:03:45 +00:00
Jacek Sieka 6cfa28e1f5 fix finalization cleanup
* add block pool finalization test
2020-01-21 15:53:53 +02:00
Zahary Karadjov 74fc34eef2 Restore some constants to pass the constants & presets test 2020-01-21 01:26:57 +02:00
Zahary Karadjov 095b271bd4 Implement a loader for custom const presets
The loader has been tested with the presets published by Lighthouse.
You can try connecting to one of their testnets by running:

cd nim-beacon-chain
./connect-to-testnet lighthouse/testnet0
2020-01-21 01:26:57 +02:00
Jacek Sieka 2a67ac3c05 trie -> kv store
* simplify data storage to key-value, tries are not relevant for NBC
* locked-down version of lmdb dependency
* easier to build / maintain on various platforms
2020-01-20 13:39:37 +00:00
Jacek Sieka d853d7c024
remove SigKey.init (unused, fixes #664) 2020-01-18 10:22:17 +01:00
Dustin Brody c301e5d42f mark 49 0.9.2/0.9.4 spec functions/data types as 0.10.0 2020-01-15 12:34:16 +00:00
Jacek Sieka 860be026e1 fix block pool init head selection
the head state is not necessarily saved in the database, so we need to
make sure we update things to the correct place
2020-01-15 12:17:49 +00:00
Mamy André-Ratsimbazafy b49003988a Add Epoch Processing to nbench 2020-01-15 10:03:56 +00:00
Dustin Brody 8eaebf7163 update to spec version 0.10.0 2020-01-14 18:17:16 +00:00
Dustin Brody 46de726e3d mark all compatible 0.9.3 functions and data structures as 0.9.4 2019-12-20 17:24:42 +00:00
Mamy Ratsimbazafy 106352aff3
Nbench - Flexible benchmarking of Nimbus internals (#641)
* nbench PoC

* Remove the yaml files from the example scenarios

* update README with current status

* Add an alternative implementation that uses defer

* Forgot to add the old proc body

* slots-processing

* allow benching state_transition failures

* Add Attestations processing (workaround confutils bug:
- https://github.com/status-im/nim-confutils/issues/10
- https://github.com/status-im/nim-confutils/issues/11
- https://github.com/status-im/nim-confutils/issues/12

* Add CLI command in the readme

* Filter report and add notes about CPU cycles

* Report averages

* Add debugecho style time/cycle print

* Report when we skip BLS and state root verification

* Update to 0.9.3

* Generalize scenario parsing

* Support all block processing scenarios

* parallel bench runner PoC

* gitBetter load issues reporting (the load issues were invalid signature and expected to fail)
2019-12-20 17:14:43 +01:00
Dustin Brody 417f96213d fix genesis interop by fixing deposit signature handling; add TRACE-level compilation to CI 2019-12-20 15:26:48 +00:00
Jacek Sieka 746659bdc6
bump stew, fix warnings (#655) 2019-12-20 14:25:33 +01:00
Dustin Brody ea4afd7454 mark 19 more 0.9.2-compatible entities as 0.9.3-compatible 2019-12-19 11:46:30 +00:00
Dustin Brody 350a6c071a mark a couple compatible dozen 0.9.2 functions/data structures as 0.9.3 2019-12-18 16:13:39 +00:00
Dustin Brody 0328cd49bf mark several more functions and data structures as 0.9.3 2019-12-18 10:00:17 +00:00
Dustin Brody 1610981be6 mark several more spec-referencing code/data parts as 0.9.3 2019-12-17 16:51:46 +00:00
Dustin Brody 27bf8908a9 mark a couple dozen functions and data structures as 0.9.3; fix TRACE-level compiling; stop downloading 0.9.2 test vectors; update cosmetic/display spec versions in test suite 2019-12-17 16:51:46 +00:00
Dustin Brody c824416f56 initial 0.9.3 spec update 2019-12-17 00:06:03 +02:00
Dustin Brody c4daa726b5 fix AggregateAndProof definition 2019-12-13 09:04:30 +00:00
Dustin Brody 294db6aaca initially implement specified attestation aggregation functions and data structures 2019-12-13 09:04:30 +00:00
Jacek Sieka a3145f0f5c cleanups
* move stuff out of beaconnode
* fix use-after-free in query
2019-12-10 14:47:52 +00:00
Zahary Karadjov 889031453f The beacon node can be compiled with the LibP2P native back-end
This hasn't been tested yet even in local sim.
2019-12-10 15:02:16 +02:00
Jacek Sieka eea523a89e logging fixes
* don't log out-of-sync when there's no work to do
* small logging fixes
2019-12-10 10:18:30 +00:00
Zahary Karadjov 8ab0248209
Revert "tighten a few imports"
This reverts commit b24a9200d9.
2019-12-10 02:18:47 +02:00
Jacek Sieka b24a9200d9 tighten a few imports 2019-12-09 12:58:28 +00:00
Dustin Brody c762c0232d render get_beacon_proposer_index(...) return value an option and remove overly aggressive assertion formerly causing crashes when missing validators (#617)
* render get_beacon_proposer_index(...) return value an option and remove overly aggressive assertion formerly causing crashes when missing validators

* follow addBlock(...) after refactoring

* [skip ci] fix typo
2019-12-04 11:49:59 +01:00
kdeme 3b3a2b10f1 First attempt to have some fuzz tests exported for the fuzzing framework 2019-12-03 14:33:47 +02:00
Dustin Brody e88e559caf render visible gradual validator emptying on catchup 2019-12-02 18:24:30 +02:00
Jacek Sieka 2a728d9468 fix stack overflow in isAncestor recursion 2019-11-28 17:36:07 +02:00
Dustin Brody bde20436e8 ~50% speedup by fast-pathing Eth2Digest merkle hashing 2019-11-26 15:23:27 +00:00
Zahary Karadjov 8de4d38e4e More tracing 2019-11-26 02:45:25 +02:00
Zahary Karadjov 957645d752 Fix the bulds with json logging enabled 2019-11-26 02:45:25 +02:00
Yuriy Glukhov 777b3f4e29 State/block pruning 2019-11-26 02:45:25 +02:00
Dustin Brody 1494bcc262 replace links to Apache and MIT licenses with HTTPS versions (#592) 2019-11-25 16:30:02 +01:00
Dustin Brody b82328b148 convert hash_tree_roots_prefix(...) to an iterator 2019-11-25 14:48:59 +00:00
Dustin Brody f6de87b80e remove O(n^2) eth1 deposit processing and scaffolding comparing it with O(n) deposit processing; re-enable now-viable-to-run genesis state initialization tests with around 16,384 validators 2019-11-25 14:48:59 +00:00
Dustin Brody fa2a703a17 switch more funcs and procs from global to module scope; create scaffolding for non-O(n^2) eth1 deposit processing with assertions for equivalent functionality; fix a few more shellcheck warnings 2019-11-25 14:48:59 +00:00
Dustin Brody a8de94ad05 add validator_leaving logging at debug chronicles level for all call paths to initiate_validator_exit(...) 2019-11-25 09:08:15 +00:00
Dustin Brody 8d9fdb0afd miscellanous cleanups and removing a few funcs/procs from unnecessary global scopes 2019-11-25 09:08:15 +00:00
Mamy Ratsimbazafy 1938379bcd
Automated beacon constant checks (#583)
* Fix SSZ bitlist deserialization super silly bug

* Add an automated sanity checks of the beacon chain constants

* Remove SSZ consensus skipping procs [skip ci]

* Add phase 1 domains

* Fix mainnet constants

* Add missing phase 1 constants on minimal (they are not needed somehow on mainnet)

* Rebase artifact: constants were defined twice
2019-11-22 20:56:39 +01:00
Dustin Brody edfd65fd5d stop download 0.9.1 test vectors; finish 0.9.2 update, primarily via fixing preset constants; remove a couple 0.19.6 shims 2019-11-22 16:41:43 +00:00
Dustin Brody f5b8931fad switch assert(...) to doAssert(...) and minor cleanup 2019-11-22 16:39:05 +00:00
Dustin Brody 41fa377638 mark all non-changed 0.9.1 items as 0.9.2 (#576) 2019-11-22 09:29:04 +01:00
Jacek Sieka 7c6d87051a
simplify validator api (#573)
* avoid passing whole state to validator api (easier when it goes
out-of-process)
* clean up some cruft
2019-11-21 10:57:59 +01:00
Dustin Brody c0f009908c
re-enable test_interop based on zcli with 0.9.1 specs and update initialize_beacon_state_from_eth1(...) to 0.9.1 (#569)
* re-enable test_interop based on zcli with 0.9.1 specs and update initialize_beacon_state_from_eth1(...) to 0.9.1

* switch many procs to funcs

* fix import os.nim instead; ospaths is deprecated [Deprecated] warnings
2019-11-21 09:15:10 +00:00
Mamy Ratsimbazafy bb0979f232
Ssz fixes (#566)
* SSZ signature from EF are always opaque blobs (security issue - https://github.com/status-im/nim-beacon-chain/issues/555)
Enable
- Attestation
- Beaconstate (minimal only)
- Deposit
- DepositData
- ProposerSlashing

Updates #518

* mv debug_ssz to helpers

* Small reorg of the list types

* Fix IndexedAttestation, AttesterSlashing and BeaconBlock

* Deactivate on mainnet: AttesterSlashing, BeaconBlockBody, IndexedAttestation, Attestation, BeaconBlock

* Fix Validators on minimal and mainnet
2019-11-19 12:04:51 +01:00
Dustin Brody 2749cb14a6 convert mostly-uncompiled verify_merkle_branch(...) calls to newer is_valid_merkle_branch(...) name; re-enable several lines of deposit testing code and outline roadmap for completion 2019-11-18 17:35:58 +00:00
Dustin Brody 6f87c8fd89 verify that state_sim is justifying and finalizing; fix 3 more warnings; rename crosslink_committee_cache to beacon_committee_cache; fix O(n^2) usage of get_base_reward(...) 2019-11-18 17:35:58 +00:00
Dustin Brody c7c439ac61
use bounded List[] SSZ type for more hashing consistency with spec 2019-11-18 09:35:17 +01:00
Jacek Sieka 14712bbbdc
random warning fixes 2019-11-15 12:05:18 +01:00
Dustin Brody 7e70d16da9
remove BeaconStateNew and associated testing scaffolding/infrastructure 2019-11-14 20:03:08 +01:00
Dustin Brody 96dcb1cf95
Remove shard-relating code 2019-11-14 19:48:12 +01:00
Dustin Brody c08698bffe
scaffolding to remove last vestiges of get_committee_count(...), SHARD_COUNT, and rest of that 0.8.x infrastructure; this demonstrates equivalence between two, before discarding old code 2019-11-14 18:37:51 +01:00
Dustin Brody df1d00f13a remove UnusedImport spam (#549)
* remove UnusedImport spam

* re-add random import to beacon_node
2019-11-14 11:47:55 +01:00
Dustin Brody 3dcc00779a replace deprecated toSet(...) with toHashSet(...); replace implicit with more explicit default constructor for readability 2019-11-13 13:49:38 +00:00
Dustin Brody 5a54c823d8 update to 0.9.1: don't try to run removed tests; re-add 1.5 tests (one only in minimal); remove bls_verify_multiple(...) and AttestationDataAndCustodyBit; and update process_attester_slashing(...), get_indexed_attestation(...), and is_valid_indexed_attestation(...) 2019-11-13 13:49:38 +00:00
Dustin Brody 1f352bf440
fix trace compilation 2019-11-12 16:35:12 +01:00
Dustin Brody fb5d9edb7c keep types consistent 2019-11-12 14:49:26 +00:00
Dustin Brody d5ce142511 fix network sim finalization; remove get_attestation_data_slot(...); remove 2 more get_crosslink_committee(...) calls 2019-11-12 14:49:26 +00:00
Zahary Karadjov 22591deced Safer testnet restarts; Working CLI queries for inspecting the genesis states
When the connect_to_testnet script is invoked it will first verify that
the genesis file of the testnet hasn't changed. If it has changed, any
previously created database associated with the testnet will be erased.

To facilitate this, the genesis file of each network is written to the
data folder of the beacon node. The beacon node will refuse to start if
it detects a discrepancy between the data folder and any state snapshot
specified on the command-line.

Since the testnet sharing spec requires us to use SSZ snapshots, the Json
support is now phased out. To help with the transition and to preserve the
functionality of the multinet scripts, the beacon node now supports a CLI
query command that can extract any data from the genesis state. This is
based on new developments in the SSZ navigators.
2019-11-11 23:29:36 +00:00
Dustin Brody d457dfee81 switch all but 4 remaining callers to get_crosslink_committee(...) to get_beacon_committee(...) 2019-11-11 11:46:03 +00:00
Dustin Brody 568fa1dcb9 mark functions, constants, and data structures unchanged by 0.9.1 as such 2019-11-11 10:25:57 +00:00
Dustin Brody 949d735155 fix remaining test 0.9.0 test fixtures and switch back to non-transitiontest setup; remove get_committee_count(...); keep SHARD_COUNT consistent with MAX_COMMITTEES_PER_SLOT 2019-11-11 10:25:57 +00:00
Dustin Brody cb4a86756b update get_seed(...) and get_beacon_proposer_index(...) to 0.9.0, implement compute_proposer_index(...), and render 3 more test fixtures working (#532)
* update get_seed(...) and get_beacon_proposer_index(...) to 0.9.0, implement compute_proposer_index(...), and render 3 more test fixtures working

* rm stray Crosslink reference which prevented static SSZ tests from building

* remove references to removed tests in attestations test fixture; add minimal-preset block sanity test, plus all but one of mainnet tests for block sanity to transition fixtures
2019-11-10 01:03:41 +01:00
Dustin Brody e984b2c932 mark is_valid_merkle_branch(...), process_justification_and_finalization(...), and process_rewards_and_penalties(...) as 0.9.0; update check_attestation(...) to 0.9.0; add 11 working test fixtures, having updated SSZ fixture directory to 0.9.0 2019-11-08 13:23:08 +00:00
Dustin Brody 275fb22c00 flip targeted spec version and ssz test fixtures to 0.9.0; almost completely update BeaconState to 0.9.0; update minimal and mainnet preset constants to 0.9.0; update process_proposer_slashing(...), get_attestation_deltas(...), and get_matching_head_attestations(...) to 0.9.0; mark process_final_updates(...) as 0.9.0 2019-11-08 13:23:08 +00:00
Dustin Brody 3bb95f571f remove remaining Crosslink and CompactCommittee vestiges 2019-11-08 13:23:08 +00:00
Dustin Brody 61c2cf9415 rm get_compact_committees_root(...), Transfer, and references to 0.9-removed BeaconState.{active_index_roots,compact_committees_roots; temporarily disable genesis interop test pending 0.9 BeaconState 2019-11-08 13:23:08 +00:00
Dustin Brody 68654848cb update get_attesting_indices(...) to 0.9.0; remove Crosslink from AttestationData to update AttestationData to 0.9.0; rm get_winning_crosslink_and_attesting_indices(...) and get_crosslink_deltas(...) 2019-11-08 13:23:08 +00:00
Dustin Brody cb5454c4eb double-check that one can back-infer shard/epoch from slot/committee-index, in preparation for removing crosslink from AttestationData 2019-11-08 13:23:08 +00:00
Dustin Brody 13e662c8dd remove scaffolding and make get_crosslink_committee(...) a purer wrapper around get_beacon_committee(...) 2019-11-08 13:23:08 +00:00
Dustin Brody c2ebe482a5 temporarily disable SSZ tests, which fail when trying to read AttestationData or structures which embed it; implemenet get_committee_count_at_slot(...) and get_beacon_committee(...); express get_crosslink_committee(...) in terms of get_beacon_committee(...) and ensure that results are identical 2019-11-08 13:23:08 +00:00
Dustin Brody a1f5f3512b
add sanity check for SIGFPE crashes (#524) 2019-11-06 15:50:12 +00:00
Dustin Brody 0073555752
transition deposit operation fixtures to 0.9.0 (#515)
* transition deposit operations fixture to 0.9.0

* mark slash_validator(...) as 0.9.0

* switch remaining non-ref objects to ref objects to maybe avoid crashes in CI

* remove unused helpers/debug_state imports
2019-11-06 15:02:06 +00:00
Dustin Brody 63e621c27d
initial 0.9.0 spec sync (#509)
* rename compute_epoch_of_slot(...) to compute_epoch_at_slot(...)

* remove some unnecessary imports; remove some crosslink-related code and tests; complete renaming of compute_epoch_of_slot(...) to compute_epoch_at_slot(...)

* rm more transfer-related code and tests; rm more unnecessary strutils imports

* rm remaining unused imports

* remove useless get_empty_per_epoch_cache(...)/compute_start_slot_of_epoch(...) calls

* rename compute_start_slot_of_epoch(...) to compute_start_slot_at_epoch(...)

* rename ACTIVATION_EXIT_DELAY to MAX_SEED_LOOKAHEAD

* update domain types to 0.9.0

* mark AttesterSlashing, IndexedAttestation, AttestationDataAndCustodyBit, DepositData, BeaconBlockHeader, Fork, integer_squareroot(...), and process_voluntary_exit(...) as 0.9.0

* mark increase_balance(...), decrease_balance(...), get_block_root(...), CheckPoint, Deposit, PendingAttestation, HistoricalBatch, is_active_validator(...), and is_slashable_attestation_data(...) as 0.9.0

* mark compute_activation_exit_epoch(...), bls_verify(...), Validator, get_active_validator_indices(...), get_current_epoch(...), get_total_active_balance(...), and get_previous_epoch(...) as 0.9.0

* mark get_block_root_at_slot(...), ProposerSlashing, get_domain(...), VoluntaryExit, mainnet preset Gwei values, minimal preset max operations, process_block_header(...), and is_slashable_validator(...) as 0.9.0

* mark makeWithdrawalCredentials(...), get_validator_churn_limit(...), get_total_balance(...), is_valid_indexed_attestation(...), bls_aggregate_pubkeys(...), initial genesis value/constants, Attestation, get_randao_mix(...), mainnet preset max operations per block constants, minimal preset Gwei values and time parameters, process_eth1_data(...), get_shuffled_seq(...), compute_committee(...), and process_slots(...) as 0.9.0; partially update get_indexed_attestation(...) to 0.9.0 by removing crosslink refs and associated tests

* mark initiate_validator_exit(...), process_registry_updates(...), BeaconBlock, Eth1Data, compute_domain(...), process_randao(...), process_attester_slashing(...), get_base_reward(...), and process_slot(...) as 0.9.0
2019-10-30 19:41:19 +00:00
Zahary Karadjov dfe3a6f0fb
Switch to Nim v1.0.2 2019-10-28 15:00:25 +02:00
Jacek Sieka ee5d1c1467
minimize imports, specially for spec - cuts 2/3 of ncli build time (#500)
* minimize imports, specially for spec - cuts 2/3 of ncli build time
* ptr_arith->ptrops
2019-10-25 12:59:56 +02:00
Dustin Brody 700e7d921b
update to beacon chain phase 0 spec version 0.8.4; no substantive changes (#494) 2019-10-24 13:36:36 +00:00
Dustin Brody 3da4c02bb3 implement beacon_current_live_validators and beacon_previous_live_validators metrics 2019-10-21 20:58:08 +09:00
Jacek Sieka b7f9d9e4be fix BlsValue inits 2019-10-08 23:52:50 +09:00
Zahary Karadjov c23b011c77 Cosmetic improvements to the status bar; Attempt to fix the CI 2019-10-07 23:56:34 +09:00
Dustin Brody 4c0b2a9a32
implement beacon_pending_deposits, beacon_processed_deposits_total, beacon_previous_validators, and beacon_current_validators metrics (#478)
* implement additional beacon_current_validators and beacon_previous_validators metrics

* tentatively implement beacon_pending_deposits additional metric

* implement beacon_processed_deposits_total additional metric
2019-10-06 04:31:50 +00:00
Dustin Brody 1a62c122d8
Finish remaining required interop metrics (#477)
* implement beacon_finalized_root, beacon_current_justified_root, and beacon_previous_justified_root interop metrics
2019-10-03 18:16:25 +00:00
Dustin Brody 255980c9f6 fix issue #367 and remove too-small range type usage for ValidatorIndex 2019-10-03 16:41:25 +03:00
Dustin Brody 2122bb44c9
implement beacon_finalized_epoch, beacon_current_justified_epoch, and beacon_previous_justified_epoch (#476) 2019-10-03 10:36:31 +00:00
Jacek Sieka f111a6a444 fix attestation being added twice to block (fixes #361) 2019-10-02 12:27:51 +03:00
Jacek Sieka b8ddb776ca fix attestation selection when calculating rewards and penalties
Previously, the we'd use the first source attestation regardless if it
matches the filter or not leading to the wrong minimum being used
2019-10-02 12:17:02 +03:00
Zahary Karadjov 806836714a Add an accidentally forgotten fix 2019-10-01 17:50:13 +03:00
Eugene Kabanov 13bde5aee3 Add gossipsub messages monitoring tool. (#458)
* Add gossipsub messages monitoring tool.

* Add support of bootstrap nodes file.

* Move topic constants to spec/network.nim.

* Add ability to monitor custom topics.
2019-10-01 15:52:28 +02:00
Mamy Ratsimbazafy 48d22d53dc
Ensure that we use the interop deposit signatures (#467)
* Ensure that we use the interop deposit signatures

* Add all interop keys to the test
2019-10-01 15:44:38 +02:00
Zahary Karadjov c39d4f1817 Fix an overflow in the justification bits processing 2019-09-27 23:29:58 +03:00
Mamy André-Ratsimbazafy 90b20fab52 Domain is now array[8, byte] instead of uint64 2019-09-26 17:53:59 +03:00
Mamy Ratsimbazafy a7a1f78499
Fix TRACE chronicles_log_level (#451) 2019-09-23 17:00:10 +02:00
Mamy Ratsimbazafy b100ceef56
Add detailed finalization and justification debug logs (#449)
* Add detailed finalization and justification debug logs

* log was wrongfully reporting rule 123 instead of rule 12
2019-09-23 15:48:26 +02:00
Mamy Ratsimbazafy 988a13b872 Official operations - deposits unit test (#408)
* [Test] Official operations - deposits unit test

* Allow ignoring deposits with invalid signature

* We need stacktraces to debug windows 64 issue https://github.com/status-im/nim-beacon-chain/pull/408#issuecomment-529236359

* fix naming of unit test proc

* Revert "We need stacktraces to debug windows 64 issue https://github.com/status-im/nim-beacon-chain/pull/408#issuecomment-529236359"

This reverts commit 04b8b05162.

* skip windows-64 in CI #435

* proposer slashing started to crash as well on win-64 #435
2019-09-11 21:10:54 +00:00
Mamy Ratsimbazafy 3dc2b87e6a Transfer - split process_transfer/processTransfers + tests + fixes (#422)
* Prepare test suite for transfers

* split API process_transfer / processTransfers

* Add range checks on transfer

* Fix invalid transfer conditions

* don't test on windows 64-bit #435
2019-09-11 20:29:00 +00:00
Mamy Ratsimbazafy 8676bbf388 Add proposer slashing tests (#431)
* Add proposer slashing tests

* typo in import
2019-09-11 07:50:07 +00:00
Mamy Ratsimbazafy 6be2e24a8f
Attester slashing - split process_attester_slashing/processAttesterSlashings + tests (#424)
* attester slashing tests - pending #415

* split process_attester_slashing/processAttesterSlashings

* Add logs to attester_slashing

* deactivate bls tests for now (https://github.com/status-im/nim-beacon-chain/issues/429) and cherry-pick from 60f2437dd2
2019-09-10 18:03:06 -04:00
Dustin Brody f72de67f63 Proposer slashing is 0.8.3 (#425)
* minimal refactor of proposer slashings to 0.8.3; no semantic difference

* actually mark spec
2019-09-10 11:29:46 -04:00
Mamy Ratsimbazafy 81b47f35d1 Voluntary exit - split single + multiple exits proc & tests (#421)
* add test suite for voluntary exit

* update API to process_voluntary_exit

* Add range check of validator_index for voluntary exits

* Revert to dual single + multiple voluntary exits API + enable in test suite

* no cache or mocking needed
2019-09-10 00:14:03 +00:00
Mamy Ratsimbazafy 061b6e0ddf
Test block headers (#418)
* Block headers tests

* Skip invalid_parent_root test due to https://github.com/status-im/nim-beacon-chain/issues/407

* Fix bls_verify with invalid sig. Activate block header tests
2019-09-09 14:40:59 -04:00
Yuriy Glukhov d4c480ff5f
Eth1 validators (#393) 2019-09-09 18:59:02 +03:00
Dustin Brody a4b4c89539 remove incorrect exit condition in get_attestation_details(...) (#416)
* remove incorrect exit condition in get_attestation_details(...)

* enable historical batch unit test, which now passes
2019-09-09 09:10:03 -04:00
Mamy Ratsimbazafy 3f446e6383
Attestations EF tests + Proper Real/Opaque BlsValue deserialization (#410)
* Add attestation unit test

* process_attestation doesn't throw exceptions

* Allow SSZ deserialization of both real and invalid signatures

* Add new process_attestation checks - pass all process_attestation tests

* Add sanity check for #361

* Fix SSZ testing after fromBytes/fromSSZBytes changes
2019-09-08 23:33:24 -04:00
Jacek Sieka 5f58af2112
fix ssz list, protocol names (#405) 2019-09-08 16:54:31 +02:00
Mamy Ratsimbazafy 82b9e008d6
Fix genesis blockheader zero signature (supercedes #395) (#400) 2019-09-07 20:56:24 -04:00
Dustin Brody 0c174036a5 update get_attesting_indices(...) to 0.8.3; mark IndexedAttestation, Attestation, PendingAttestation, and get_randao_mix(...) as 0.8.3; rm duplicate/dead code get_unslashed_attesting_indices(...) (#391) 2019-09-06 21:58:38 +02:00
Yuriy Glukhov e3bd4410d8
Revert "Eth1 validators" (#392)
This reverts commit bb8955bdbc.
2019-09-06 13:00:56 +03:00
Yuriy Glukhov bb8955bdbc Eth1 validators 2019-09-06 02:38:38 +02:00
Dustin Brody f882c92a42 fix network sim (#389)
* fix network sim

* mark BeaconState, state list/vector lengths, misc values, get_base_reward(...), verifyStateRoot(...), and process_slot(...) as 0.8.3; update minimal/mainnet config initial values to 0.8.3 by removing GENESIS_FORK_VERSION
2019-09-05 21:52:34 +02:00
Jacek Sieka 31db5d3a62 Deserialize serialized default bls values (#388)
Ugly workaround to quick-fix broken network/interop sim
2019-09-05 14:27:28 +00:00
zah 7ebf685fa3 Work-around for #373 (#385) 2019-09-05 10:27:26 +00:00
Mamy Ratsimbazafy ad240953ed Fix #378, int to bytes32, LATEST_RANDAO_MIXES, get_seed, get_crosslink_committee, get_compact_committee_root (#380) 2019-09-04 13:57:18 +00:00
Dustin Brody b9265e15f1
remove some superfluous/pointless spec version markers (#377) 2019-09-04 09:25:27 +00:00
Mamy Ratsimbazafy c11f37e550 V0.8.3 tests part 3 - sanity checks for slots and blocks (#375)
* Add sanity check for slot processing (also impacted by https://github.com/status-im/nim-beacon-chain/issues/373)

* use reportDiff also for all state tests vs EF

* initial sanity checks for blocks - workaround zero signature in block headers: https://github.com/status-im/nim-beacon-chain/issues/374

* Remove generic object variant compare commented code

* Add the one block state transition sanity checks

* generalize blocks test to multiple blocks

* simplify slots test runner

* Add official epoch transitions, sanity blocks, sanity slots to the test suite

* Fix index out-of-bounds in initiate_validator_exit - enable proposer slashings unittest
2019-09-03 18:02:21 +00:00
Mamy Ratsimbazafy 297e9079d4 V0.8.3 tests part2 - BLS + Shuffling + State transition epoch (#372)
* Update BLS fixtures to 0.8.3

* Bump fixtures with shuffling with minimal preset

* Update shuffling tests

* parseTest generic over file format (Json or SSZ)

* Initial crosslink parsing commit for debugging Nim crash

* Workaround https://github.com/status-im/nim-beacon-chain/issues/369

* Crosslink test works for minimal - https://github.com/status-im/nim-beacon-chain/issues/369 is back on mainnet

* Use ref objects to workaround https://github.com/status-im/nim-beacon-chain/issues/369

* Generalize state transition epoch test to all epoch tests

* Fix slashing (potential uint64 overflow in previous spec)

* Add a state debugging macro to deeply inspect the wrong fields

* make reportDiff visible

* Improve the debug state macro for containers
2019-09-03 03:12:09 +00:00
Jacek Sieka 6d798c821f log json errors when loading deposits 2019-09-02 22:14:18 +03:00
Jacek Sieka 9cfbdb5e16 more interop fixes
* fix eth1 interop block hash
* update initialize_beacon_state_from_eth1 to 0.8.3
2019-09-02 22:14:18 +03:00
Jacek Sieka 7b73b40bab interop updates
* add interop launcher scripts
* stick validator_keygen into beacon_node
* fix lmd ghost slot number on missing block
* use mocked eth1data when producing blocks
* use bls public key method for withdrawal credentials
* fix deposit domain
* prefer lowercase for a bunch of toHex
* build simulation binary in data folder to avoid data types confusion
2019-09-02 22:14:18 +03:00
Mamy Ratsimbazafy ca4f29caca v0.8.1 tests refactor (#326)
* Introduce new mocking proc to replace:
- makeFakeValidatorPrivKey
- hackPrivKey
- getNextBeaconProposerIndex
- addBlock
- makeBlock

* Add comments on datastructure unsynced with the spec

* Add merkle tree constructor and initial mocking for deposits (missing merkle proofs)

* [Mock] Implement sparse merkle tree and merkle proof builder

* [Mocking] Genesis deposits

* Add compact_committees_roots init + mock genesis state

* [Tests] Add first deposit test using the new mocking procedures

* [Tests -deposits] add at and over 32 ETH deposit tests

* [Tests - deposits] Add test for validator top-up

* [Tests -deposits] Mention the TODO to test for invalid conditions

* [Tests] Add stub to test "is_valid_genesis_state"

* [Merkle proofs] Implement round-trip checks

* Deactivate roundtrips test

* SSZ - use EF convention for hash_tree_root / hashTreeRoot

* [Tests - Attestation] Attestation mocking + initial test

* Add mocking + 3 new tests for valid attestations + mention future invalid attestation tests

* Add crosslinks test (1 failing to attestations in block being duplicated in state transition)

* Single attestation crosslink test - workaround https://github.com/status-im/nim-beacon-chain/issues/361

* Add test for failed crosslink penalty

* Rebase fixes + add refactored tests to test suite

* justif-finalization helpers first batch

* Add 234 finalization tests

* Fix justif test, Rule I 234 finalization does not happen with sufficient support.
(Also unittest check template does not fail properly in some cases)

* Add tests for all finalization rules

* Properly delete nim-byteutils following c91727e7e5 (diff-7c3613dba5171cb6027c67835dd3b9d4)

* use digest helper for deposit root
2019-08-28 12:07:00 +00:00
Bruno Škvorc a7998f5c3e
Update datatypes.nim 2019-08-27 15:03:00 +02:00
Bruno Škvorc 5254ec0e7f
Add assert message
Tripped me up during build, error was unnecessarily unspecific.
2019-08-27 14:59:57 +02:00
Dustin Brody 8665efbbd1 bump beacon chain versions to 0.8.3, which has no relevant changes in this regard (#358) 2019-08-26 11:09:47 +02:00
Dustin Brody cc0429a643
slightly update/rename verify_merkle_branch(...) to 0.8.2 is_valid_merkle_branch(...); mark check_attestation(...), get_active_validator_indices(...), get_seed(...), and BeaconBlockBody as 0.8.2; update compute_domain(...) 0.8.2 by reversing field order (#356) 2019-08-26 07:57:09 +00:00
Dustin Brody c47e3903be rm some less-useful debug and tracing noise (#355) 2019-08-23 11:39:23 +02:00
Zahary Karadjov 912563648e
Fix the json build; Don't long the non-attesting indices in the testnets 2019-08-21 14:21:08 +03:00
Jacek Sieka 78f7a4a505 restore network sim finalization
* relax attestation validation when attestation is incoming but make it
more strict when adding to block
* share attestation validation logic between attestation pool and state
transition
* remove a bunch of redundant logging
* fix potential underflow in attestation delay checking
* fix committee used for attestation shard selection when attesting
* fix attestation data construction
2019-08-19 21:05:54 +03:00
Jacek Sieka 225465578c fix startup time casts, log fixes 2019-08-19 21:05:54 +03:00
Zahary Karadjov 2fca9465e7
Fix the build with RLPx 2019-08-15 20:43:05 +02:00
Jacek Sieka e639cd95c7
humaneXNum -> shortLog 2019-08-15 18:01:55 +02:00
Zahary Karadjov 4daeebbf7e
Fix the build when json logging is enabled 2019-08-14 17:38:39 +02:00
Dustin Brody 3e587d4667
Justification/finalization amelioration using testnet1 shard count and epoch length (#331)
* working justification, 1 node, 64 validators

* closer to tn1 params: 256 validators

* more debugging output; switch to minimum test case

* working justification and finalization in local network simulation

* fix currently incorrect state transition/attestation test assumption
2019-08-14 08:56:32 +00:00
Dustin Brody 6704749eb0 mark unchanged 0.8.1 as 0.8.2 spec aspects (#336) 2019-08-09 14:08:00 +02:00
Zahary Karadjov a897ab9c70
Attempt to make the test suite green 2019-08-07 06:09:26 +03:00
Zahary Karadjov 5ce50b3aca
Implement the latest networking spec
https://github.com/ethereum/eth2.0-specs/pull/1328
2019-08-07 05:22:28 +03:00
Zahary Karadjov 398ea55801
Implement the latest SSZ specification and integrate the official SSZ test suite 2019-08-05 03:18:48 +03:00
Yuriy Glukhov 958e4cd3b4 Getting genesis through web3 2019-07-31 16:39:26 +03:00
Zahary Karadjov 33dfc8b2d5
Fix the compilation when json logging is used 2019-07-16 13:40:24 +03:00
Dustin Brody b049dded88
summarize why previous epoch attestations can be recapped where they are 2019-07-16 11:16:04 +02:00
Dustin Brody f6fd089ae7
Add non-attesting-validators instrumentation in process_justification_and_finalization(...) (#323)
* update process_justifications_and_finalizations(...) to 0.8.1
2019-07-16 07:34:11 +00:00
Mamy Ratsimbazafy d1d5497233
v0.7.x-v0.8.1 state transition refactor part2 (#322)
* State sim compiles and run

* Works on libp2p testnet1

* Successfully update the state transition tests

* naive update of attestation pool (failing attstations can arrive in any order)

* Remove an official assert to allow attestation reordering
2019-07-15 23:10:40 +02:00
Dustin Brody 712038fe21
mark applicable 0.8.0-compliant functions and data structures as 0.8.1 and update a couple of other comments (#321) 2019-07-15 12:57:18 +00:00
Dustin Brody c92806b950
include both static and dynamic check for invariants to ensure that get_crosslink_committee(...) can access all committees by ensure that SHARD_COUNT >= SLOTS_PER_EPOCH (#318)
* include both static and dynamic check for invariants to ensure that get_crosslink_committee(...) can access all committees by ensure that SHARD_COUNT >= SLOTS_PER_EPOCH

* add state_sim to CI, to ensure that it also doesn't break SHARD_COUNT/SLOTS_PER_EPOCH assumptions, along with just in general not breaking
2019-07-13 15:58:01 +00:00
Dustin Brody 7bc2b81e18
~2x state_sim speedup via additional caching in get_crosslink_committee(...) (#316) 2019-07-10 14:00:28 +00:00
Dustin Brody dfa062db7d 0.8.0, continued (#315)
* rename get_genesis_beacon_state(...) to initialize_beacon_state_from_eth1(...); remove unused vestiges get_temporary_block_header(...) and get_empty_block(...); partly align initialize_beacon_state_from_eth1(...) with 0.8.0 version; implement CompactCommittee object type; update process_final_updates(...) to 0.8.0

* mark get_shard_delta(...) as 0.8.0

* mark get_total_active_balance(...) and epoch transition helper functions as 0.8.0

* move FORK_CHOICE_BALANCE_INCREMENT, not in spec anymore, to sole user in fork_choice
2019-07-10 14:23:02 +02:00
Dustin Brody 0a2d09e7fe
Speed up process_crosslinks(...) and get_crosslink_deltas(...) by 10x - 15x in state_sim (#314)
* remove some low-hanging perf silliness from get_winning_crosslink_and_attesting_indices(...) and switch inner/outer loop nesting order to get 10-15x function speedup for 128 and 512 validator cases by avoiding accidentally quadratic behavior, while keeping function signature unchanged and allowing easy ongoing verification of correctness of optimization

* avoid memory allocation with an openarray rather than seq

* update get_crosslink_deltas(...) to 0.8.0

* move non-spec ValidatorSetDeltaFlags from datatypes.nim to sync_protocol; fix mainnet MIN_ATTESTATION_INCLUSION_DELAY preset; update get_attestation_deltas(...) to 0.8.0; for 512 validators, get 90x speedup for get_attestation_deltas(...) from 179s to 2s

* explain purpose of when false portions, so they can be removed once accomplished
2019-07-08 13:19:52 +00:00
Jacek Sieka b9e62a1f6c
std_shims -> stew 2019-07-07 11:53:58 +02:00
Dustin Brody 68be98cafa
More 0.8.0 updates (#311)
* replace BeaconState.finalized_{epoch,root} with BeaconState.finalized_checkpoint; rename get_delayed_activation_exit_epoch(...) to compute_activation_exit_epoch(...) and mark as 0.8.0; update get_churn_limit(...)/get_validator_churn_limit(...) to 0.8.0; update process_registry_updates(...) to 0.8.0

* update process_crosslinks(...) to 0.8.0; mark compute_start_slot_of_epoch(...) and get_committee_count(...) as 0.8.0

* mark Fork, is_slashable_validator(...), and get_beacon_proposer_index(...) as 0.8.0

* rename LATEST_SLASHED_EXIT_LENGTH to EPOCHS_PER_SLASHINGS_VECTOR; update process_slashings(...) to 0.8.0; remove pointless type conversion warning in get_previous_epoch(...)

* convert remaining references to finalized_epoch to finalized_checkpoint.epoch

* update slash_validator(...) to 0.8.0; mark inital value, Gwei, and time constants as 0.8.0; mark hash(...) and processBlockHeader(...) as 0.8.0

* rename WHISTLEBLOWING_REWARD_QUOTIENT to WHISTLEBLOWER_REWARD_QUOTIENT; rename LATEST_ACTIVE_INDEX_ROOTS_LENGTH to EPOCHS_PER_HISTORICAL_VECTOR (randao will also get merged into this); remove get_active_index_root(...); mark time parameter, signature domain types, and max operations per block constants as 0.8.0; update rewards and penalties constants to 0.8.0

* update is_valid_indexed_attestation(...) to 0.8.0; mark process_slot(...) as 0.8.0

* replace BeaconState.{current,previous}_justified_{epoch,root} with BeaconState.{current,previous}_justified_checkpoint
2019-07-05 08:30:05 +00:00
Dustin Brody 5198930503 add start_epoch consistent with rest of crosslink 2019-07-03 10:41:42 +03:00
Dustin Brody 8a5e5334d6 re-implement attestation checking in 0.8.0 form and rename relevant function from checkAttestation(...) to spec's process_attestation(...), though there's a slight difference in how they're factored; remove a couple of pointless type conversion warnings; rename validate_indexed_attestation(...) to is_valid_indexed_attestation(...); remove verify_bitfield(...) not in 0.8.0; update AttestationData to 0.8.0 by using Checkpoint data structure; rename MAX_CROSSLINK_EPOCHS to MAX_EPOCHS_PER_CROSSLINK 2019-07-03 10:41:42 +03:00
Dustin Brody d7905351eb update get_crosslink_committee(...) to 0.8.0 and fix https://github.com/status-im/nim-beacon-chain/issues/307 2019-07-02 19:52:39 +03:00
Dustin Brody 95dea4c165 mark get_attestation_data_slot(...), ProposerSlashing, AttesterSlashing, DepositData, BeaconBlock, Eth1Data, AttestationDataAndCustodyBit, and is_slashable_attestation_data(...) as 0.8.0; add Checkpoint object type 2019-07-01 14:44:57 +03:00
Dustin Brody fa39eb82c4 do most of 0.8.0 BeaconState updating, in particular field renamings, re-orderings, etc; rm now-unnecessary genesis block creation aspects, towards the near-stub version in current spec as mostfields are 0/default/etc values 2019-07-01 14:44:57 +03:00
Dustin Brody fb601bd5b8 rename BeaconState.validator_registry to BeaconState.validators 2019-07-01 14:44:57 +03:00
Dustin Brody 2ba14bc9af update get_previous_epoch(...) to 0.8.0 and mark compute_committee(...) as 0.8.0 2019-07-01 14:44:57 +03:00
Dustin Brody dc98ffb09b update Validator to 0.8.0; update/refactor deposit contract/etc to constants category; mark process_rewards_and_penalties(...) as 0.8.0 2019-07-01 14:44:57 +03:00
Dustin Brody 607a7de5eb rename get_epoch_start_shard(...) to 0.8.0 get_start_shard(...) 2019-07-01 14:44:57 +03:00
Dustin Brody e4321dc4ed Initial frozen phase 0/v0.8.0 spec (#305)
* rename slot_to_epoch(...) to compute_epoch_of_slot(...)

* rename aggregation_bitfield/custody_bitfield fields to aggregation_bits/custody_bits; rename convert_to_indexed(...) to get_indexed_attestation(...); mark BeaconBlockHeader as 0.8.0; update minimal preset MIN_ATTESTATION_INCLUSION_DELAY/time parameters in general to 0.8

* fix beacon node compilation; it used slightly different capitalizations of slot_to_epoch/slotToEpoch

* mark integer_squareroot(...), Deposit, VoluntaryExit, and Transfer as 0.8.0; rename get_epoch_start_slot(...) to compute_start_slot_of_epoch(...)

* add new Domain alias for uint64; rename bls_domain(...) to compute_domain(...), MAX_INDICES_PER_ATTESTATION to MAX_VALIDATORS_PER_COMMITTEE, and SignatureDomain to DomainType; update get_domain(...) to 0.8.0

* update/mark initiate_validator_exit(...) and Crosslink as 0.8.0; rename BeaconState.latest_block_roots -> BeaconState.block_roots; mark HistoricalBatch as 0.8.0
2019-07-01 09:53:42 +02:00
Dustin Brody 7356905f95
remove get_epoch_start_slot(...) kludge and update to work exactly as 0.7.1 specifies; remove unused get_attestation_participants_cached(...); update AttestationData to 0.7.1 (being the last data structure, I believe, remaining for such; remove potentially spurious/certainly not-in-spec assertion from compute_committee(...); fix state sim to work with new get_epoch_start_slot/AttestationData/etc setup where it can't stuff all shards' attestations from same slot in the same MIN_ATTESTATION_INCLUSION_DELAY rotating/circular buffer of seq[Attestation]s without more involved shuffling of shard/slot calculation order; fix attestation pool testing to be consistent with get_epoch_start_slot(...) (#302) 2019-06-29 09:17:24 +00:00
Mamy Ratsimbazafy 214860cb88 Split state_transition (#301) 2019-06-28 13:44:44 +00:00
Dustin Brody 43a9e904cf
replace get_attestation_slot(...) with 0.7.1 get_attestation_data_slot(...) and update all callers; update AttestationData to be almost 0.7.1, except nonextistent slot field; update Crosslink and get_winning_crosslink_and_attesting_indices(...) to 0.7.1 (#297) 2019-06-25 11:17:08 +00:00
Dustin Brody c7e06374f4
Remove get_crosslink_committees_at_slot and fix research/state_sim (#291)
* migrate attestation pool tests from get_crosslink_committees_at_slot(...) to get_crosslink_committee(...)

* rm obsolete, unused get_crosslink_committees_at_slot_cached(...) and migrate tests/test_state_transition from get_crosslink_committees_at_slot(...) to get_crosslink_committee(...)

* migrate tests/testutil from get_crosslink_committees_at_slot(...) to get_crosslink_committee(...)

* use more pervasive caching infrastructure, initially of compute_committee; remove buggy (and per-index) shuffling to fix research/state_sim, which was noticing validators on multiple shard committees; rm now-unused specific get_attesting_balance_cached

* add some get_active_validator_index caching

* rm obsolete/unused get_attesting_indices_cached

* rm get_crosslink_committees_at_slot(...)

* some more 0.7 changes -- some of which can't be completed pending some data structure updates -- and shard handling changes to offset with epoch start shards
2019-06-24 09:21:56 +00:00
Dustin Brody 13381c8fea
update everything https://github.com/ethereum/eth2.0-specs/releases/tag/v0.7.1 leaves unchanged from v0.7.0 to v0.7.1 (#290) 2019-06-18 15:34:30 +00:00
Dustin Brody e9180c490e
more 0.7.0 updates (#287)
* update Deposit and BeaconBlockHeader to 0.7.0

* update get_total_balance(...) and get_base_reward(...) to 0.7.0

* rename initial value constant BLS_WITHDRAWAL_PREFIX_BYTE to BLS_WITHDRAWAL_PREFIX

* mark minimal presets as entirely migrated to 0.7.0; update process_rewards_and_penalties(...) to 0.7.0

* specify that targeting 0.7.0; adopt 0.7.0 refactoring of bls_domain/get_domain to update get_domain(...) to 0.7.0; mark that mainnet constant presets and block header processing are 0.7.0-consistent; update process_justification_and_finalization to 0.7.0 to fix potential underflow errors

* address mratsim's comment to copy whole spec comment
2019-06-14 16:21:04 +00:00
Dustin Brody 2a6e64d3f9
0.7.0 updates including semantic changes (#285)
* update get_previous_epoch(...)

* update BeaconBlock to 0.7.0 by renaming previous_block_root field to parent_root
2019-06-14 13:50:47 +00:00
Dustin Brody e2496567b3 More safe 0.7.0 updates (#283)
* complete marking unchanged-in-0.7.0 beaconstate functions

* complete marking unchanged-in-0.7.0 spec refs functions in helpers.nim and time.nim: get_randao_mix(...), bytes_to_int(...), and description of beacon chain processing apropos time

* complete remaining feasible 0.7.0 updates which lack externally visible effect
2019-06-14 09:50:14 +02:00
Dustin Brody 6357bc2a98 0.7.0 updates which don't change semantics; mostly comment changes (#281)
* 0.7.0 updates which don't change semantics; mostly comment changes

* mark get_attesting_indices, bls_verify, ProposerSlashing, BeaconBlockBody, deposit contract constants, state list length constants, compute_committee, get_crosslink_committee, is_slashable_attestation_data, processAttesterSlashings as 0.7.0; rename BASE_REWARD_QUOTIENT to BASE_REWARD_FACTOR

* complete marking unchanged-in-0.7.0 datatypes as such; a few more have trivial field renamings, etc
2019-06-13 09:40:58 +02:00
Dustin Brody d400650eeb
0.6.3 updates (#279)
* update IndexedAttestation, verify_slashable_attestation/verify_indexed_attestation, and attester slashing processing to 0.6.3

* rm debug scaffolding

* rename Attestation.aggregate_signature -> Attestation.signature; convert various references to AttestationData.slot to get_attestation_slot; implement convert_to_indexed; update checkAttestation and processAttestations to 0.6.3; remove spurious assertion in beacon node related to invalid attestations

* replace 0.5 get_winning_root_and_participants with 0.6 get_winning_crosslink_and_attesting_indices; update process_crosslinks to 0.6.3

* mark both remaining 0.6.0 spec implementations as 0.6.3

* clear out all remaining spec version 0.6.1 refs

* GENESIS_SLOT and GENESIS_EPOCH are 0

* rm 0.5 get_attestation_participants in favor of 0.6 get_attesting_indices

* address mratsim's comment

* time can be equal to genesis

* fix invalid block assertions; those were essentially spurious

* allow toBeaconTime to handle time before genesis (in accordance with now(...) which states time can exist before GENESIS)
2019-06-12 07:48:49 +00:00
Yuriy Glukhov 10c7920b27 Proto changes to facilitate backward sync (#271)
* Proto changes to facilitate backward sync

* Update to latest spec types in sync proto

* Use blockpool for more straightforward block headers collection

* Added BlockPool.getRef

* Update beacon_chain/sync_protocol.nim

Co-Authored-By: Jacek Sieka <arnetheduck@gmail.com>
2019-06-10 07:13:53 -04:00
Dustin Brody 252819a84d
bump everything that's not one of the 3 or so specific changes in 0.6.3 from 0.6.2 to 0.6.3 if already at 0.6.2 (#277) 2019-06-05 10:25:30 +00:00
Dustin Brody 2e1515b107
0.6.2 updates (#275)
* update process_justification_and_finalization to 0.6.2; mark AttesterSlashing as 0.6.2

* replace get_effective_balance(...) with state.validator_registry[idx].effective_balance; rm get_effective_balance, process_ejections, should_update_validator_registry, update_validator_registry, and update_registry_and_shuffling_data; update get_total_balance to 0.6.2; implement process_registry_updates

* rm exit_validator; implement is_slashable_attestation_data; partly update processAttesterSlashings

* mark HistoricalBatch and Eth1Data as 0.6.2; implement get_shard_delta(...); replace 0.5 finish_epoch_update with 0.6 process_final_updates

* mark increase_balance, decrease_balance, get_delayed_activation_exit_epoch, bls_aggregate_pubkeys, bls_verify_multiple, Attestation, Transfer, slot_to_epoch, Crosslink, get_current_epoch, int_to_bytes*, various constants, processEth1Data, processTransfers, and verifyStateRoot as 0.6.2; rm is_double_vote and is_surround_vote

* mark get_bitfield_bit, verify_bitfield, ProposerSlashing, DepositData, VoluntaryExit, PendingAttestation, Fork, integer_squareroot, get_epoch_start_slot, is_active_validator, generate_seed, some constants to 0.6.2; rename MIN_PENALTY_QUOTIENT to MIN_SLASHING_PENALTY_QUOTIENT

* rm get_previous_total_balance, get_current_epoch_boundary_attestations, get_previous_epoch_boundary_attestations, and get_previous_epoch_matching_head_attestations

* update BeaconState to 0.6.2; simplify legacy get_crosslink_committees_at_slot infrastructure a bit by noting that registry_change is always false; reimplment 0.5 get_crosslink_committees_at_slot in terms of 0.6 get_crosslink_committee

* mark process_deposit(...), get_block_root_at_slot(...), get_block_root(...), Deposit, BeaconBlockHeader, BeaconBlockBody, hash(...), get_active_index_root(...), various constants, get_shard_delta(...), get_epoch_start_shard(...), get_crosslink_committee(...), processRandao(...), processVoluntaryExits(...), cacheState(...) as 0.6.2

* rm removed-since-0.5 split(...), is_power_of_2(...), get_shuffling(...); rm 0.5 versions of get_active_validator_indices and get_epoch_committee_count; add a few tests for integer_squareroot

* mark bytes_to_int(...) and advanceState(...) as 0.6.2

* rm 0.5 get_attesting_indices; update get_attesting_balance to 0.6.2

* another tiny commit to poke AppVeyor to maybe not timeout at connecting to GitHub partway through CI: mark get_churn_limit(...), initiate_validator_exit(...), and Validator as 0.6.2

* mark get_attestation_slot(...), AttestationDataAndCustodyBit, and BeaconBlock as 0.6.2
2019-06-03 10:31:04 +00:00
Jacek Sieka 16fb9e8d11 Add support for dealing with overlapping attestations (#262) 2019-06-03 08:26:38 +00:00
Dustin Brody 6a47ca3c46
More 0.6.1 updates (#274)
* implement get_churn_limit from 0.6.1; update initiate_validator_exit and get_previous_epoch to 0.6.1; remove obsoleted/non-spec reduce_balance, replaced by decrease_balance; mark BeaconBlock as 0.6.1

* rename get_block_root to get_block_root_at_slot and introduce epoch-oriented get_block_root; implement get_attestation_slot, get_matching_source_attestations, get_matching_target_attestations, get_matching_head_attestations, 0.6.1 get_attesting_indices, get_unslashed_attesting_indices, get_attestation_deltas, process_rewards_and_penalties; rm get_inactivity_penalty

* update Validator and processVoluntaryExits to 0.6.1; rm obsolete inclusion_slots/inclusion_distances

* rm removed activate_validator; mark DepositData, misc values, Gwei values, Randao processing, state caching as 0.6.1; update GENESIS_SLOT to 64, since it doesn't quite yet work at 0

* mark BeaconBlockHeader as 0.6.1; update BeaconBlockBody to 0.6.1; rename WHISTLEBLOWER_REWARD_QUOTIENT to WHISTLEBLOWING_REWARD_QUOTIENT; update slash_validator to 0.6.1

* implement 0.6.2 is_slashable_validator; update processProposerSlashings to 0.6.2; mark state caching as 0.6.2

* mark get_total_active_balance and process_slashings as 0.6.2; update get_base_reward to 0.6.2

* rm prepare_validator_for_withdrawal, process_exit_queue; mark mainnet misc constants as 0.6.2; update process block header to 0.6.2

* address mratsim's code review comment
2019-05-29 10:08:56 +00:00
Mamy Ratsimbazafy 6b1f0e816c
Introduce constant presets (#269)
* Initial move constants to mainnet preset

* Bump fixtures to include shuffling with minimal preset

* Add minimal preset with shuffling test

* Add minimal preset to the full test suite

* use preset() everywhere
2019-05-27 14:48:13 +02:00
Dustin Brody 46cfce652d
initial new get_beacon_proposer_index and dependencies (get_shuffled_index, get_shard_delta, get_epoch_start_shard, get_crosslink_committee) (#266)
* initial new get_beacon_proposer_index and dependencies (get_shuffled_index, get_shard_delta, get_epoch_start_shard, get_crosslink_committee)

* more 0.6.1 updates

* clean up debugEchos and fix validator keygen to create correct indices
2019-05-23 11:13:02 +00:00
Mamy Ratsimbazafy e5047d93a4 [Tests] SHA256 + official shuffling vectors (#263)
* Change digests to SHA2-256 (from Keccak256)

* Fix sha256 digest initialization - pass shuffling test

* Add comments + remove old shuffling tests

* Remove stale tree hashing tests

* Small toOpenArray optim + notes on in-place hashing

* Revert "Revert "Update test repo""

This reverts commit f385467cd3.

* Revert "Revert "Fix SIGFPE on shuffling for 0""

This reverts commit 226d380f87.

* Revert "Revert "Implement shuffling test (pending typedesc and shuffling algo fix)""

This reverts commit 813cb6fbb8.

* withEth2hash templates now needs ctx.init()

* Use init-update-finish to avoid burnMem
2019-05-10 08:14:01 +00:00
Dustin Brody d977c96ae1 0.6.1 - signing_root, deposits, validator indices and committee count (#261)
* signed_root -> signing_root

* implement new process_deposit; update timing parameters to 0.6.1; update Deposit to 0.6.1 and remove DepositInput

* update get_active_validator_indices and get_epoch_committee_count to 0.6.1; rm get_current_epoch_committee_count, get_previous_epoch_committee_count, and get_next_epoch_committee_count; bump state_sim default validator count a bit more

* re-introduce 0.5.1-ish get_active_validator_indices, get_epoch_committee_count as scaffolding for still-0.5.1ish shuffling
2019-05-09 14:27:37 +02:00
Mamy André-Ratsimbazafy 226d380f87
Revert "Fix SIGFPE on shuffling for 0"
This reverts commit 8cfbfdf565.
2019-05-09 12:07:09 +02:00
Mamy André-Ratsimbazafy 8cfbfdf565
Fix SIGFPE on shuffling for 0 2019-05-09 12:01:01 +02:00
Jacek Sieka 17fa499755 cache state root a bit more aggressively (#260)
needed to be able to run state sim, too slow otherwise
2019-05-04 14:10:45 +00:00
Jacek Sieka bd4893d2e8 head block selection fixes (#259)
* track justified and head blocks correctly in presence of blank slots
* fix new block state root
* keep list of viable heads instead of walking children
* disable json bitfield deserializer (needs a corresponding serializer)
* fix handshake best & finalized root
2019-05-01 12:19:29 +03:00
Dustin Brody 46b4154ce8 begin 0.6.0 (#256)
* begin 0.6.0: new get_domain/increase_balance/reduce_balance, BeaconState.validator_balances -> BeaconState.balances, some renamed constants, transaction processing changes, SlashableAttestation field name changes, 0.6.0 get_beacon_proposer_index always uses given state's slot, update tests subrepo

* mark get_bitfield_bit/bls_verify_multiple/stat-list-lengths/is_active_validator/is_surround_vote/slot_to_epoch/int_to_bytes/etc as unchanged in 0.6.0; rm Eth1DataVote/maybe_reset_eth1_period and thus adjust expected tree hash test results

* mark verify_bitfield/bls_verify/deposit-contract/VoluntaryExit/PendingAttestation/Historicalbatch/Fork as 0.6.0; update DOMAIN_BEACON_BLOCK to DOMAIN_BEACON_PROPOSER

* update Crosslink to 0.6.0 (also requires tree hashing test result change, so isolate in individual commit)

* mark verify_merkle_branch/get_delayed_activation_exit_epoch/ProposerSlashing/Attestation/AttestationDataAndCustodyBit/hash/integer_squareroot/get_epoch_start_slot/is_double_vote/get_randao_mix/generate_seed as 0.6.0; update reward and penalty quotients; SlashableAttestation -> IndexedAttestation; rm get_fork_version; ATTESTATION_INCLUSION_REWARD_QUOTIENT -> PROPOSER_REWARD_QUOTIENT
2019-04-29 10:48:30 -06:00
Mamy Ratsimbazafy 5784b2d7f7
Full EF state tests parsing (#242)
* Load full state test + Add json-serialization for Bitfield

* Implement empty_block_transition test (but test is missing the end state)

* Use the provided empty block instead of mocking one

* Add failing block signing test

* Tests that can't be passed now are now "for information" + indent the "information"/hash given
2019-04-29 18:10:01 +02:00
Jacek Sieka 9a2f3176f7 guard node.state (#237)
* fix node.state being used unsafely across async callbacks (fixes #230)
* node.state -> node.stateCache
* fix attestation validation not to use asserts
* re-enable attestation pool tests
* prepare for resizing validator registry
2019-04-06 09:46:07 +02:00
Dustin Brody c53de3e550 Caching updates/refactoring & state_sim defaults updates (#235)
* rm now-superceded shuffling cache (shuffling is only called from get_crosslinks), which was badly architected due to trying to exist in state; rm one more vestige of previous light-client regime (one more to go, from datatypes)

* fix wrong shuffling list size (active validator size, not validator size) to make consistent with 0.5.1 (will be inconsistent with testnet0); fix typo and change defaults in state_sim

* doAssert a couple of constant relationships necessary to avoid underflow; rm non-spec, unused helper function get_new_recent_block_roots

* refactor separate crosslink_committee_cache and winning_root_participants_cache(s) into StateData object; remove last vestige of previous shuffling cache

* separate out caching parts of StateData to new StateCache object
2019-04-05 08:18:13 -06:00
Jacek Sieka 605dd0a0e9 Some speedups (#226)
* ssz: avoid memory allocations

* a bit fishy with the 32-item stack.. this should be a smallvector

* digest: avoid another burnmem

* avoid a few allocations by using iterator
2019-04-03 15:46:22 +00:00
cheatfate 3117a4481e Remove unneeded initialization. 2019-03-28 23:24:50 +02:00
cheatfate 6f4c7b775c Remove one-line hash to avoid burnMem.
Remove unnecessary copyMem()s.
2019-03-28 23:24:50 +02:00
Jacek Sieka 201b079fe3 avoid bad blocks by filtering included attestations
* fix epoch violation during initial attestation validation
* fill in attestation fields (finalizes again!)
2019-03-28 21:10:08 +02:00
Dustin Brody 2c566df411 rm debug message 2019-03-28 10:40:20 +02:00
Dustin Brody ced4dbe859 4x speedup on epoch processing for 64 validators 2019-03-28 10:40:20 +02:00
Jacek Sieka 3ad6053298 Use block, slot tuple to idenfiy fork
this is the beginning of tracking block-slots more precisely, so we can
the justification epoch slot bug.

* avoid asyncDiscard (swallows assertions)
* fix attestation delay
* fix several state root cache bugs
* introduce workaround for genesis epoch spec issue
2019-03-28 10:32:09 +02:00
Zahary Karadjov 406df9d44f More chatty syncing 2019-03-27 18:13:39 +02:00
Jacek Sieka 81c66fbce0
use signed_root as canonical block root (#211)
* no need to pass prev_block_root when updating state
* some Slot fixes
* fix `hash_tree_root` for Slot, Epoch
* detect missing hash_tree_root type support
* remove some <0.5 state checks
2019-03-26 19:32:35 -06:00
Mamy André-Ratsimbazafy 8273345a2c fix comment 2019-03-26 17:35:09 +02:00
Mamy André-Ratsimbazafy 8b5e18b7a4 Magic fix to #209 2019-03-26 17:35:09 +02:00
Mamy André-Ratsimbazafy 57c693c88f Only check for epoch underflow just before peer communication
During state processing epochs can underflow (hence why we have a non-zero arbitrary genesis epoch)
2019-03-26 17:35:09 +02:00
Mamy André-Ratsimbazafy e189c1d3cd Crash before making attestation when replaying state pass genesis 2019-03-26 17:35:09 +02:00
Mamy André-Ratsimbazafy e8160c2126 more logs to investigate epoch underflow/overflow https://github.com/status-im/nim-beacon-chain/issues/207 2019-03-26 17:35:09 +02:00
Jacek Sieka 1b0e67c88c
ssz: update to 0.5.1:ish (#202)
* ssz: update to 0.5.1:ish
* slightly fewer seq allocations
* still a lot of potential for optimization
* fixes #174
* ssz: avoid reallocating leaves (logN merkle impl)
2019-03-25 10:46:31 -06:00
Dustin Brody 6b1527ccd5 Almost final 0.5.1 spec updates (#201)
* update processExits to 0.5.1

* mark process_deposit, get_empty_block as 0.5.1; update get_genesis_beacon_state to 0.5.1; implement processDeposits

* hotfix via https://github.com/ethereum/eth2.0-specs/pull/821
2019-03-24 12:21:23 -06:00
Dustin Brody 9f55b4646c
More 0.5.1 spec updates (#195)
* rm gone-in-0.5.0 Proposal, verifyBlockSignature, and slot check which moved to spec function processBlockHeader

* mark get_attestation_participants and get_epoch_committee_count as 0.5.1; finish updating processAttestations to 0.5.1; add kludgy workaround for bug relating to get_winning_roots_etc using crosslink_data_root as index when we have that as ZERO_HASH for all leading to it confusing attesters on different shards; rm BeaconState.latest_attestations, which splits into previous_epoch_attestations and current_epoch_attestations

* Fix CI due to removed latest_attestations field
2019-03-22 18:33:12 +00:00
Zahary Karadjov 0a027e410a Address review concerns and make some steps towards json logging 2019-03-22 17:04:17 +02:00
Dustin Brody 94c3b005fa
https://github.com/ethereum/eth2.0-specs/pull/816 hotfix (#194) 2019-03-21 16:02:50 +00:00
Dustin Brody 1bb2859aca
implement bulk of 0.5.0 state transition function (#192) 2019-03-21 15:31:36 +00:00
Jacek Sieka ee89ef1c79
cleanups (#189)
* add simple bitfield type (fixes #19)
* fix bit endianess to match spec
* consolidate attestation and block logging
* cruft cleanup
* run optimizer on tests, speeds up build
2019-03-20 14:01:48 -06:00
Dustin Brody 13c7f7fcec begin refactoring 0.5.0 epoch transition (#185)
* begin refactoring 0.5.0 epoch transition at justification and finalization

* update processTransfers to 0.5.0 spec

* update get_crosslink_committees_at_slot to 0.5.0 except for noted workaround for pre-0.5.0 spec bug related to epoch boundary conditions, to be flipped in more coordinated way with other epoch processing changes

* mark ejection processing as 0.5.0

* 0.5.0 epoch update finalization

* rm BEACON_CHAIN_SHARD_NUMBER to complete updating miscellaneous constants to 0.5.0

* mark verify_slashable_attestation as 0.5.0

* update process_attester_slashing to 0.5.0

* update process_slashings(...) to 0.5.0 and mark process_exit_queue as 0.5.0

* refactor epoch processing by implementing 0.5.0's update_registry_and_shuffling_data(...)
2019-03-19 20:19:46 -06:00
Dustin Brody 2bc632c29b
AttestationData.justified epoch -> AttestationData.source_epoch and AttestationData.latest_crosslink -> AttestationData.previous_crosslink; implement verify_merkle_branch and update process_deposit to use it; rm BeaconState.batched_block_roots; Transfer.{from, to} -> Transfer.{sender, recipient}; update Fork data type and code to use bytes4; implement int_to_bytes8 (close to metaprogramming time); adjust signed_root and all callers to use 0.5.0 calling signature; processDepositRoot -> processEth1Data (#182) 2019-03-18 15:42:42 +00:00
Dustin Brody 798197879b
Revert "AttestationData.justified epoch -> AttestationData.source_epoch and AttestationData.latest_crosslink -> AttestationData.previous_crosslink; implement verify_merkle_branch and update process_deposit to use it; rm BeaconState.batched_block_roots; Transfer.{from, to} -> Transfer.{sender, recipient}; update Fork data type and code to use bytes4; implement int_to_bytes8 (close to metaprogramming time); adjust signed_root and all callers to use 0.5.0 calling signature; processDepositRoot -> processEth1Data"
This reverts commit 18f1d000a1.
2019-03-17 15:24:50 -07:00
Dustin Brody 18f1d000a1
AttestationData.justified epoch -> AttestationData.source_epoch and AttestationData.latest_crosslink -> AttestationData.previous_crosslink; implement verify_merkle_branch and update process_deposit to use it; rm BeaconState.batched_block_roots; Transfer.{from, to} -> Transfer.{sender, recipient}; update Fork data type and code to use bytes4; implement int_to_bytes8 (close to metaprogramming time); adjust signed_root and all callers to use 0.5.0 calling signature; processDepositRoot -> processEth1Data 2019-03-17 15:17:40 -07:00
Dustin Brody f36c2d86dc begin 0.5.0 spec update (#179)
* begin 0.5.0 spec update: parent_root -> previous_block_root, BeaconState.justified_epoch -> BeaconState.current_justified_epoch, DOMAIN_PROPOSAL -> DOMAIN_BEACON_BLOCK, temporarily rename BeaconBlockHeader to BeaconBlockHeaderRLP to allow for gradual re-merging without disrupting RLP; mark a few unchanged functions and data types, implement get_temporary_block_header/get_empty_block/should_update_validator_registry/processBlockHeader/cacheState; update a few others

* a dozen or so more trivial mostly comment changes, finding more unchanged parts of 0.5.0

* several more trivial changes; goal is to reduce noise around the upcoming substantial changes (epoch processing, etc)
2019-03-16 13:52:37 -06:00
Jacek Sieka 1cb8ae9004
state processing fixes (#177)
* remove some redundant state updates
* when attesting late, use correct state / head
* don't send out obsolete attestations
* don't propose obsolete blocks
* remove some more resundant state updates :)
* simplify block logging (experimental)
* document fork choice division
* fix some Slot / Epoch conversion warnings
2019-03-14 07:33:56 -06:00
Ștefan Talpalaru de295619be
assert() -> doAssert() 2019-03-14 00:04:43 +01:00
Zahary Karadjov 562eafe124 Finish the Spec/Epoch types refactoring 2019-03-13 23:23:01 +02:00
Dustin Brody 1479bae22f implement distinct Epoch type and continue 0.4.0 spec update (#173)
* implement distinct Epoch type
* update two more spec 0.4.0 markers, leaving just deposit processing
2019-03-12 19:46:44 -06:00
Dustin Brody 50a0948bc0 Mostly convert to using distinct Slot type (#172) 2019-03-12 16:21:32 -06:00
Dustin Brody 434ba5727f finish all but 3 items for 0.4.0 spec update (#167) 2019-03-12 10:59:43 +01:00
Dustin Brody 13de015038 more 0.4.0 spec updates: renamings, continue Proposal updates, and note several more functions which were verified not to have changed in 0.4.0 (#163) 2019-03-11 09:33:24 -06:00
Dustin Brody 89aa3ec53f
fix keccak256 digest comments 2019-03-09 12:34:08 -08:00
Jacek Sieka aec31b1126 Updates for finalizing chain! (#162)
* set epoch_boundary_root - chain finalizes!
* fix slotStart to offset GENESIS_SLOT
* work around bug that will be fixed by
https://github.com/ethereum/eth2.0-specs/pull/732
* compile with debug info (there was a GC-related crash in C land)
2019-03-09 04:23:14 +00:00
Dustin Brody 0b47acae37
some comment changes noting further 0.4.0-spec-update progress, but no meaningful semantic changes (#161) 2019-03-08 18:23:42 +00:00
Dustin Brody 1afdcda62d
spec version 0.4.0 update: shard_block_root -> crosslink_data_root; switch from validator.slashed_epoch -> validator.slashed approach; a couple assert -> doAssert; note fix for underflow-prone spec changes in checkAttestation; GENESIS_SLOT change from 2^63 to 2^32; refactor Proposer and ProposalSlashing structure; add signed_root; remove duplicate slashValidator (#159)
* spec version 0.4.0 update: shard_block_root -> crosslink_data_root; switch from validator.slashed_epoch -> validator.slashed approach; a couple assert -> doAssert; note fix for underflow-prone spec changes in checkAttestation; GENESIS_SLOT change from 2^63 to 2^32; refactor Proposer and ProposalSlashing structure; add signed_root; remove duplicate slashValidator

* re-apply shard_block_root -> crosslink_data_root

* remove incorrect humaneSlotNum

* add (run-time only, alas) sanity check on signed_root
2019-03-08 17:44:31 +00:00
Dustin Brody 72749f4d04
processEpoch 5x faster, total 2x faster, with 16k validators (#157)
* processEpoch 5x faster, total 2x faster, with 16k validators

* don't recompute active_validator_indices as much

* remove duplicate process_ejections function
2019-03-07 16:53:44 +00:00
Dustin Brody 5d0de00168 transform a few quadratic algorithms from processEpoch into linear or otherwise subquadratic algorithms (#155) 2019-03-06 08:21:13 -06:00
Dustin Brody b8402e9809
speed up epoch processing by caching shuffling results (#153)
* speed up epoch processing by caching shuffling results
2019-03-05 19:10:36 +00:00
Yuriy Glukhov 21e4debda0
Merge pull request #117 from status-im/sync2
Initial full sync impl
2019-03-04 21:05:24 +02:00
Yuriy Glukhov 9ea0bf4326
Comments addressed 2019-03-04 13:50:26 +02:00
Dustin Brody 81d5becc7b Switch hash functions, prevent underflow & verify that shuffling works (#149)
* prevent balance underflow, per spec implementation note

* verify that permutation shuffing works: add YAML dependency to be used for ingesting test cases; switch from blake2 to keccak256 to match EF test cases; remove inefficient naive-spec-version of shuffling algorithm now that protolambda's can be tested directly
2019-03-01 17:50:01 -06:00
Yuriy Glukhov bed6510da3
Initial full sync impl 2019-03-01 19:39:38 +02:00
Zahary Karadjov 68a39a21be Mysterious fix for the mysterious build issue 2019-03-01 18:51:37 +02:00
Dustin Brody 794f6aed89
fix https://github.com/status-im/nim-beacon-chain/issues/145 (#147) 2019-03-01 14:39:17 +00:00
Jacek Sieka ddb3a9cd94
Revert "Add some missing RLP serialization routines (#144)" (#146)
This reverts commit 859aa327cb.
2019-03-01 07:34:37 -06:00
zah 859aa327cb Add some missing RLP serialization routines (#144)
* Add some missing RLP serialization routines
2019-02-28 15:50:21 -06:00
Dustin Brody 1cfea240ed Complete scan through v0.3.0 spec (#141)
* Complete scan through v0.3.0 spec; remaining known discreptancy is in bls_verify_multiple

* switch remaining slot-misuses of get_active_validator_indices to epochs

* spec workaround

* finish first 0.3.0 pass by getting slow, naive version of bls_verify_multiple running
2019-02-28 15:24:43 -06:00
Jacek Sieka 125231d321
add initial block pool (#139)
* implement in-memory block graph
* store tail block in database
* resolve unknown parents by syncing them from peers
* introduce concept of resolved blocks and attestations - those that
follow minimal protocol rules
* update state head lazily
* log more stuff
* shortHash -> shortLog
* start 9/10 beacon nodes by default, last can be started manually
* see also #134
* fix start.sh epoch length
2019-02-28 15:21:29 -06:00
Mamy Ratsimbazafy 20f99db058 remove unused epoch template (#142) 2019-02-28 08:17:57 -06:00
Dustin Brody 34e6149f89
fix crosslink epoch refs to begin with GENESIS_EPOCH; more minor epoch processing tweaks/unstubbed-code (#137)
* fix crosslink epoch refs to begin with GENESIS_EPOCH; more minor epoch processing tweaks/unstubbed-code

* remove obsolete TODO and use humanEpochNum
2019-02-27 13:58:07 +00:00
Dustin Brody 0305432a87 implement and call process_slashings and process_exit_queue from processEpoch; update a few more epoch processing details to be consistent with 0.3.0; add rest of final upates per epoch (#136) 2019-02-26 10:54:08 +01:00
Dustin Brody 60e6a9fedc assert relationship between constants necessary to avoid underflow 2019-02-25 14:58:04 +00:00
Dustin Brody eda43b5b1c adjust copyright comment form 2019-02-25 14:58:04 +00:00
Dustin Brody 6dab867848 cleanup 2019-02-25 14:58:04 +00:00
Dustin Brody 8baab40ea6 much faster shuffling based on observations from https://github.com/protolambda/eth2-shuffle enough that it's not easy to get exact speedup numbers 2019-02-25 14:58:04 +00:00
Dustin Brody de2cad0e12 switch from legacy pre-spec sum_effective_balances(...) to get_total_balance(...) 2019-02-25 14:58:04 +00:00
Dustin Brody a42601e29e initial epoch processing update switching from slot-oriented to epoch-oriented, using newer helper functions 2019-02-25 14:58:04 +00:00
Dustin Brody fb3bbfccaa small 0.3.0 adjustments; comment updates/clarifications; add Transfer processing 2019-02-25 14:58:04 +00:00
Yuriy Glukhov a62d25ab98
Merge pull request #131 from status-im/state-replay
initial state replay implementation
2019-02-22 18:04:48 +02:00
Dustin Brody 2d52d5cbfe convert some asserts to doAsserts to keep them in release mode builds; rename get_initial_beacon_state to get_genesis_beacon_state to track spec; switch target spec version to 0.3.0; switch references to penalize_validator to slash_validator/slashValidator to track spec; make some function returns safer by omitting 'return' (#132)
* convert some asserts to doAsserts to keep them in release mode builds; rename get_initial_beacon_state to get_genesis_beacon_state to track spec; switch target spec version to 0.3.0; switch references to penalize_validator to slash_validator/slashValidator to track spec; make some function returns safer by omitting 'return'

* 2x shuffling speedup by hoisting pivot calculations per https://github.com/protolambda/eth2-shuffle
2019-02-22 10:56:45 +01:00
Jacek Sieka 2d307e2257
initial state replay implementation
* fix initial attestation pool on reordered attestations
* simplify db layer api
* load head block from database on startup, then load state
  * significantly changes database format
* move subscriptions to separate proc's
* implement block replay from historical state
* avoid rescheduling epoch actions on block receipt (why?)
* make sure genesis block is created and used
* relax initial state sim parameters a bit
2019-02-20 22:42:17 -06:00
Dustin Brody 1d13007627 More spec v0.3.0 notations and updates (#129)
* tighten get_active_index_root epoch assertion bound

* implement merkle_root

* rm process_penalties_and_exits; add slash_validator

* rm rm'd and unused constant
2019-02-20 14:35:27 -06:00
Dustin Brody 132f1147a2 spec 0.3.0 updates: EPOCH_LENGTH -> SLOTS_PER_EPOCH, SlotNumber -> Slot, EpochNumber -> Epoch (#127)
* spec 0.3.0 updates: EPOCH_LENGTH -> SLOTS_PER_EPOCH, SlotNumber -> Slot, EpochNumber -> Epoch
2019-02-19 19:33:58 -06:00
Dustin Brody 1269e001ad get most 0.3.0 spec renamings (#126)
* get most 0.3.0 spec renamings
2019-02-19 17:07:56 -06:00
Jacek Sieka c64cd2f275 a few simple SlotNumber cleanups 2019-02-18 22:43:33 +02:00
Mamy Ratsimbazafy 8e48bac58b
[WIP] Fix some issues after introducing "humaneEpochNumber" (#118)
* Fix some issues after introducing "humaneEpochNumber" + use slot_duration to 4

* Address humane comments

* Fix comment
2019-02-18 16:58:34 +01:00
Jacek Sieka 0e9cc20cfe beacon node simplifications (#116)
* remove previous randao scheme code
* fix some epoch logging
* move genesis time selection to state generation
2019-02-15 17:33:32 +01:00
Dustin Brody d3f97c1d5c
Merge pull request #115 from status-im/nus
complete switching to v0.2.0 RANDAO scheme, with verification enabled
2019-02-15 10:57:14 +00:00
Dustin Brody 8fbf87aa8c complete switching to v0.2.0 RANDAO scheme, with verification enabled 2019-02-14 13:41:04 -08:00
Jacek Sieka a71424d768
add more config options and log compile parameters (#114)
default sim runs with smaller chain now
2019-02-14 13:32:33 -06:00
Mamy Ratsimbazafy c2a52d7fc5 Allow faster slots (#112)
* Allow compile time configurable slot time
2019-02-14 13:15:10 +01:00
Dustin Brody 55ac8d337b some v0.2.0 updates and fixes a bug that should enable simulation to continue longer (#110)
* some v0.2.0 updates and fixes a bug that should enable simulation to continue longer
2019-02-14 04:16:50 +01:00
Dustin Brody 31a4c7255f more v0.2.0 spec updates (#107)
* switch to v0.2.0 shuffling scheme

* fix some >80 char lines; add get_total_balance helper function (used in epoch processing, etc); note intended removal of repeat_hash and users thereof; add some int_to_bytes and bytes_to_int helpers; more underflow comments; add Gwei type; adjust generate_seed to include additional entropy source; note egregious inefficiencies

* minor code cleanup

* move some general helper functions to helpers.nim and clarify little-endian comments
2019-02-13 11:26:32 +01:00
Mamy Ratsimbazafy 899c8e5df9
Bitfield debugging asserts (#106)
* debugging asserts

* Ensure initialization of custody_bitfield for mock Attestations
2019-02-12 18:07:44 +01:00
Yuriy Glukhov 56a48cc60f Fixed epoch actions (#105)
* Fixed epoch actions

* debugging asserts

* Revert "debugging asserts"

This reverts commit 9a11f5a7d8.
2019-02-12 16:56:58 +01:00
Dustin Brody 433e2f78c2
Merge pull request #104 from status-im/oug
Spec updates for 0.2.0
2019-02-11 15:48:00 +00:00
Dustin Brody f1cbc4a9dc adjust indentation 2019-02-11 16:32:22 +01:00
Mamy Ratsimbazafy f7feb9a6c9
Delay validators and add more log in crosslink asserts (#103) 2019-02-11 16:29:21 +01:00
Dustin Brody b03ac3427f split out get_previous_epoch into helper function 2019-02-11 16:10:46 +01:00
Dustin Brody d44999abc5 latest_crosslink_root -> latest_crosslink; adjust checkAttestation; add SHUFFLE_ROUND_COUNT and FORK_CHOICE_BALANCE_INCREMENT constants; some additional casts because a few places where ValidatorIndex was used are now uint64; remove some old commented-out lines of code from shardcommittee transition 2019-02-11 15:32:27 +01:00
Dustin Brody 4747477160 more checking that various functions match v0.1; some variable/etc renaming; fix is_double_vote to check epoch, not slot equivalence; finish verify_slashable_attestation; adjust is_surround_vote logic (#100) 2019-02-08 00:07:15 +01:00
Zahary Karadjov c567bc410f
Use human-readable names for the slot numbers 2019-02-07 22:14:08 +01:00
mratsim 1e574cf75a Hacky (?) fix #97 assert fails 2019-02-07 21:22:46 +01:00
Dustin Brody 072c1607fd implement new RANDAO reveals (#92)
* implement new RANDAO reveals
2019-02-07 21:13:10 +01:00
Dustin Brody f2545318db get_crosslink_committees_at_slot updates (#95) 2019-02-07 20:27:13 +01:00
Dustin Brody 2794181889 Spec updates (#94)
* s/slot_included/inclusion_slot/; s/participation_bitfield/aggregation_bitfield/; rearrange some type definitions to match spec order

* a couple more Eth1Data-related spec syncs
2019-02-07 17:55:48 +01:00
Zahary Karadjov 31abd31d8a Fix an int conversion issue 2019-02-07 17:53:02 +01:00
Jacek Sieka 976e1e1f10
spec: remove unused code, fix state_sim speed 2019-02-07 13:03:02 +01:00
cheatfate 20e7ca48b5 Fix `validator not in registry` and `$` bugs. 2019-02-07 12:51:21 +02:00
Dustin Brody 2d71b998c7 Spec update to v0.1 (#88)
* update based on https://github.com/ethereum/eth2.0-specs/pull/483
* more spec updates
2019-02-07 11:43:21 +01:00
Dustin Brody 788b093284
Merge pull request #87 from status-im/vdy
Spec updates
2019-02-06 22:52:36 +00:00
Dustin Brody 74cb86ec21 more spec changes 2019-02-06 21:37:21 +01:00
Mamy Ratsimbazafy 224f22d377
Fix regression from #80 on validator_keygen (#86) 2019-02-06 20:58:18 +01:00
Jacek Sieka 2f96c4bade
spec: remove ShardCommittee remnants, add CrosslinkCommittee helper 2019-02-06 20:37:25 +01:00
Yuriy Glukhov 7f1bddb267 Use nim-eth (#82) 2019-02-05 20:21:18 +01:00
Dustin Brody d1af069c55 Spec update (#83)
* rm GWEI_PER_ETH

* update EJECTION_BALANCE; rm MAX_CASPER_VOTES/replace with MAX_INDICES_PER_SLASHABLE_VOTE; modify GENESIS_SLOT

* update various constants from slots to epochs; MAX_CASPER_SLASHINGS -> MAX_ATTESTER_SLASHINGS

* AttesterSlashing field renamings; SlashableVote -> SlashableAttestation; add DOMAIN_RANDAO

* rm aggregate_signature_poc_[01]_indices

* centralize some bitfield functions
2019-02-05 19:40:27 +01:00
Mamy Ratsimbazafy e5d152c6d6
Update BLS to the latest scheme. (#80)
* Attempt to switch to nim-blscurve.

* "Fix" BLS test for new tests

* Missing domain param in validator keygen
2019-02-05 17:13:29 +01:00
Dustin Brody dda24619e8 several mechanical renamings 2019-01-30 19:30:34 +02:00
Dustin Brody 68ca7850f4 rm min_empty_validator_index, ZERO_BALANCE_VALIDATOR_TTL, Validator.latest_status_change_slot; switch Fork.fork_slot to Fork.epoch; begin removing Validator.randao_layers; now that https://github.com/ethereum/eth2.0-specs/issues/352 is fixed, test out for more slots; get process_deposit consistent with current spec 2019-01-30 00:41:07 -08:00
Dustin Brody 8dadbd3257 fix spurious assertion per https://github.com/ethereum/eth2.0-specs/issues/515 2019-01-28 20:49:20 -08:00
Dustin Brody 5d0e1c026d remove Validator.{custody_commitment,latest_custody_reseed_slot,penultimate_custody_reseed_slot} 2019-01-28 20:34:44 -08:00
Dustin Brody ad3de5fde9 Uint24 -> ValidatorIndex; Fork.pre_fork_version -> Fork.previous_version; Fork.post_fork_version -> Fork.current_version 2019-01-28 20:15:00 -08:00
Dustin Brody 4574178836 epochs replace slots as main first-class entity; several mechanical renamings (ForkData -> Fork; .fork_data -> .fork; remove stub custody types/code; some type aliasing to keep slots and epochs more enforceably separate; fix test_state_transition.nim assumption that GENESIS_SLOT == 0 2019-01-28 19:22:22 -08:00
Dustin Brody b46183047f remove BeaconState.shard_committees_at_slots in favor of crosslink committees; validator_registry_latest_change_slot -> validator_registry_update_slot mechanical renaming 2019-01-28 23:43:55 +02:00
Dustin Brody f85088651c Revert "remove BeaconState.shard_committees_at_slots in favor of crosslink committees; validator_registry_latest_change_slot -> validator_registry_update_slot mechanical renaming"
This reverts commit 2222a8e222.
2019-01-26 11:17:22 -08:00
Dustin Brody 2222a8e222 remove BeaconState.shard_committees_at_slots in favor of crosslink committees; validator_registry_latest_change_slot -> validator_registry_update_slot mechanical renaming 2019-01-26 11:15:33 -08:00
Dustin Brody 3877ef263c remove return to gain better error messages 2019-01-23 09:53:51 -08:00
Jacek Sieka 63f7302a41
Update beacon_chain/spec/beaconstate.nim
Co-Authored-By: tersec <tersec@users.noreply.github.com>
2019-01-23 17:51:59 +00:00
Dustin Brody a11dcdfe64 switch from get_shuffling_prev to get_shuffling; remove some also-removed-in-spec calls and temporarily/ugly-bridge a still-useful reference to shard committees until they're completely removed 2019-01-22 17:50:51 -08:00
Dustin Brody 21b8bd7f71 new shuffling routine and adjust tests for it 2019-01-22 14:36:31 -08:00
Dustin Brody d728f113ac move existing shuffling routine out of way and keep existing references to it 2019-01-22 11:17:34 -08:00
Dustin Brody eed9ea8f0a Spec updates (#64)
* s/CrosslinkRecord/Crosslink/

* {MIN,MAX}_DEPOSIT to {MIN,MAX}_DEPOSIT_AMOUNT; rm SHARD_PERSISTENT_COMMITTEE_CHANGE_PERIOD; per https://github.com/ethereum/eth2.0-specs/issues/378 and 35adc9c61a rm persistent_committees, persistent_committee_reassignments, and ShardReassignmentRecord

* complete switchover from status/ValidatorStatusCodes to status_flags/INITIATED_EXIT/WITHDRAWABLE validator representation
2019-01-21 19:26:58 +01:00
Dustin Brody 476052b314 deposit root to Eth1Data transition 2019-01-18 12:47:30 +02:00
Dustin Brody c788d72857 s/POW_RECEIPT_ROOT_VOTING_PERIOD/ETH1_DATA_VOTING_PERIOD/; rm COLLECTIVE_PENALTY_CALCULATION_PERIOD, ACTIVE_PENDING_EXIT, EXITED_WITH_PENALTY 2019-01-18 12:47:30 +02:00
Dustin Brody 1fe5ba07b8 update initiate_validator_exit and exit_validator helper functions to current spec; remove spec-removed EXITED_WITHOUT_PENALTY 2019-01-18 12:47:30 +02:00
Dustin Brody 9dcf37aad6 adjust get_shuffling and get_active_validator_indices to use slot, per spec 2019-01-18 12:47:30 +02:00
Dustin Brody f30b4f822e ValidatorRecord -> Validator 2019-01-18 12:47:30 +02:00
Dustin Brody 38c2cc72fa poc -> custody mechanical name changes 2019-01-18 12:47:30 +02:00
Dustin Brody ce72c1b4bc re-disable assertion creationg failures in research/state_sim; hadn't been set up to run before, either 2019-01-17 00:14:55 -08:00