Commit Graph

312 Commits

Author SHA1 Message Date
Zahary Karadjov b454501595 handle missing validators dir during key import 2019-03-19 23:04:22 +02:00
Zahary Karadjov 42ca6afae2 Helpful scripts for resetting the testnets 2019-03-19 21:50:22 +02:00
Zahary Karadjov 6a35d3584d Automate the creation of the network metadata files
With these changes, running a simulation is very close to running
an actual testnet. Some checks have been added in the client to
make sure you are not connecting to an incompatible network (e.g.
a network running with a different number of shards).
2019-03-19 20:27:54 +02:00
Jacek Sieka 8bab6fd51f
Add TODO for local bind ip 2019-03-19 11:19:27 -06:00
Jacek Sieka d7f972a47e allow setting advertised external ip
* detect it in start.sh
2019-03-19 10:50:09 +02:00
Dustin Brody 2bc632c29b
AttestationData.justified epoch -> AttestationData.source_epoch and AttestationData.latest_crosslink -> AttestationData.previous_crosslink; implement verify_merkle_branch and update process_deposit to use it; rm BeaconState.batched_block_roots; Transfer.{from, to} -> Transfer.{sender, recipient}; update Fork data type and code to use bytes4; implement int_to_bytes8 (close to metaprogramming time); adjust signed_root and all callers to use 0.5.0 calling signature; processDepositRoot -> processEth1Data (#182) 2019-03-18 15:42:42 +00:00
Zahary Karadjov 6bb38095c9 Command-line and data storage handling for testnets 2019-03-18 12:45:29 +02:00
Dustin Brody 798197879b
Revert "AttestationData.justified epoch -> AttestationData.source_epoch and AttestationData.latest_crosslink -> AttestationData.previous_crosslink; implement verify_merkle_branch and update process_deposit to use it; rm BeaconState.batched_block_roots; Transfer.{from, to} -> Transfer.{sender, recipient}; update Fork data type and code to use bytes4; implement int_to_bytes8 (close to metaprogramming time); adjust signed_root and all callers to use 0.5.0 calling signature; processDepositRoot -> processEth1Data"
This reverts commit 18f1d000a1.
2019-03-17 15:24:50 -07:00
Dustin Brody 18f1d000a1
AttestationData.justified epoch -> AttestationData.source_epoch and AttestationData.latest_crosslink -> AttestationData.previous_crosslink; implement verify_merkle_branch and update process_deposit to use it; rm BeaconState.batched_block_roots; Transfer.{from, to} -> Transfer.{sender, recipient}; update Fork data type and code to use bytes4; implement int_to_bytes8 (close to metaprogramming time); adjust signed_root and all callers to use 0.5.0 calling signature; processDepositRoot -> processEth1Data 2019-03-17 15:17:40 -07:00
Dustin Brody f36c2d86dc begin 0.5.0 spec update (#179)
* begin 0.5.0 spec update: parent_root -> previous_block_root, BeaconState.justified_epoch -> BeaconState.current_justified_epoch, DOMAIN_PROPOSAL -> DOMAIN_BEACON_BLOCK, temporarily rename BeaconBlockHeader to BeaconBlockHeaderRLP to allow for gradual re-merging without disrupting RLP; mark a few unchanged functions and data types, implement get_temporary_block_header/get_empty_block/should_update_validator_registry/processBlockHeader/cacheState; update a few others

* a dozen or so more trivial mostly comment changes, finding more unchanged parts of 0.5.0

* several more trivial changes; goal is to reduce noise around the upcoming substantial changes (epoch processing, etc)
2019-03-16 13:52:37 -06:00
Jacek Sieka 1cb8ae9004
state processing fixes (#177)
* remove some redundant state updates
* when attesting late, use correct state / head
* don't send out obsolete attestations
* don't propose obsolete blocks
* remove some more resundant state updates :)
* simplify block logging (experimental)
* document fork choice division
* fix some Slot / Epoch conversion warnings
2019-03-14 07:33:56 -06:00
Ștefan Talpalaru 9ff1eb4ac8
Merge branch 'master' of github.com:status-im/nim-beacon-chain 2019-03-14 00:05:12 +01:00
Ștefan Talpalaru de295619be
assert() -> doAssert() 2019-03-14 00:04:43 +01:00
Jacek Sieka b0f4034060
implement fork choice (#175)
* keep track of a finalized block
* keep track of all justified blocks
* use naive spec version of LMD ghost
* cache slot number and a few more things in BlockRef
* keep track of the latest vote of each validator
* depend less on the state of node.state (it's a cache, effectively)
2019-03-13 16:59:20 -06:00
Zahary Karadjov 562eafe124 Finish the Spec/Epoch types refactoring 2019-03-13 23:23:01 +02:00
Dustin Brody 1479bae22f implement distinct Epoch type and continue 0.4.0 spec update (#173)
* implement distinct Epoch type
* update two more spec 0.4.0 markers, leaving just deposit processing
2019-03-12 19:46:44 -06:00
Dustin Brody 50a0948bc0 Mostly convert to using distinct Slot type (#172) 2019-03-12 16:21:32 -06:00
Mamy André-Ratsimbazafy 6dea18c140
Squashed commit of the following:
commit bed6909f33
Author: Mamy André-Ratsimbazafy <mamy_github@numforge.co>
Date:   Mon Mar 11 12:22:35 2019 +0100

    Cleanup imports

commit 241f62ed8d
Author: Mamy André-Ratsimbazafy <mamy_github@numforge.co>
Date:   Mon Mar 11 12:06:00 2019 +0100

    Group beacon types to prevent recursive imports
2019-03-12 16:03:14 +01:00
Zahary Karadjov 5d186ef93c Fix the build of serialized_sized 2019-03-12 15:38:58 +02:00
Zahary Karadjov 0206496663 Fix a build error 2019-03-12 15:14:30 +02:00
Zahary Karadjov 903cb8a8b5 A LibP2P-based networking stack;
To enable it, comment out the 'withLibp2p' line in nim.cfg

The history was squashed in order to remove an accidentally
commited binary file.

Other changes:

* SSZ was adapted to use the common serialization framework

* gossibsup.subscribe is not using async handlers at the moment
  and this allowed me to simplify it
2019-03-12 13:36:55 +02:00
Dustin Brody 434ba5727f finish all but 3 items for 0.4.0 spec update (#167) 2019-03-12 10:59:43 +01:00
Yuriy Glukhov b587455e7b Use BlockPool for slot->blocks mapping (#166) 2019-03-11 09:38:36 -06:00
Dustin Brody 13de015038 more 0.4.0 spec updates: renamings, continue Proposal updates, and note several more functions which were verified not to have changed in 0.4.0 (#163) 2019-03-11 09:33:24 -06:00
Dustin Brody 89aa3ec53f
fix keccak256 digest comments 2019-03-09 12:34:08 -08:00
Jacek Sieka aec31b1126 Updates for finalizing chain! (#162)
* set epoch_boundary_root - chain finalizes!
* fix slotStart to offset GENESIS_SLOT
* work around bug that will be fixed by
https://github.com/ethereum/eth2.0-specs/pull/732
* compile with debug info (there was a GC-related crash in C land)
2019-03-09 04:23:14 +00:00
Dustin Brody 0b47acae37
some comment changes noting further 0.4.0-spec-update progress, but no meaningful semantic changes (#161) 2019-03-08 18:23:42 +00:00
Dustin Brody 1afdcda62d
spec version 0.4.0 update: shard_block_root -> crosslink_data_root; switch from validator.slashed_epoch -> validator.slashed approach; a couple assert -> doAssert; note fix for underflow-prone spec changes in checkAttestation; GENESIS_SLOT change from 2^63 to 2^32; refactor Proposer and ProposalSlashing structure; add signed_root; remove duplicate slashValidator (#159)
* spec version 0.4.0 update: shard_block_root -> crosslink_data_root; switch from validator.slashed_epoch -> validator.slashed approach; a couple assert -> doAssert; note fix for underflow-prone spec changes in checkAttestation; GENESIS_SLOT change from 2^63 to 2^32; refactor Proposer and ProposalSlashing structure; add signed_root; remove duplicate slashValidator

* re-apply shard_block_root -> crosslink_data_root

* remove incorrect humaneSlotNum

* add (run-time only, alas) sanity check on signed_root
2019-03-08 17:44:31 +00:00
Jacek Sieka 6bcefc0e42
verify blocks before storing in block pool / database (#158)
* fix state db lookup typo
* fix randao reveal slot when proposing blocks
* only store blocks that can be applied to a state
* store state at every epoch boundary (yes, needs pruning!)
* split out state advancement function when there's no block
* default state sim to 0.9 attestation ratio
2019-03-08 10:40:17 -06:00
Dustin Brody 72749f4d04
processEpoch 5x faster, total 2x faster, with 16k validators (#157)
* processEpoch 5x faster, total 2x faster, with 16k validators

* don't recompute active_validator_indices as much

* remove duplicate process_ejections function
2019-03-07 16:53:44 +00:00
Jacek Sieka 4d55cf8eea beacon node sim: various improvements (fixes #111) (#156)
* allow running more or fewer validators
* use deterministic key generation for tests to avoid exhausting system
RNG
* update README with simulator docs
* write the data of each validator to separate file, instead of a big
chainstart.json (makes it easier to run different validator counts)
2019-03-07 13:59:28 +00:00
Dustin Brody 5d0de00168 transform a few quadratic algorithms from processEpoch into linear or otherwise subquadratic algorithms (#155) 2019-03-06 08:21:13 -06:00
Dustin Brody b8402e9809
speed up epoch processing by caching shuffling results (#153)
* speed up epoch processing by caching shuffling results
2019-03-05 19:10:36 +00:00
Yuriy Glukhov 38f48753b2 Comments on #117 addressed (#154) 2019-03-05 07:55:09 -06:00
Yuriy Glukhov 21e4debda0
Merge pull request #117 from status-im/sync2
Initial full sync impl
2019-03-04 21:05:24 +02:00
Yuriy Glukhov 9ea0bf4326
Comments addressed 2019-03-04 13:50:26 +02:00
Dustin Brody 81d5becc7b Switch hash functions, prevent underflow & verify that shuffling works (#149)
* prevent balance underflow, per spec implementation note

* verify that permutation shuffing works: add YAML dependency to be used for ingesting test cases; switch from blake2 to keccak256 to match EF test cases; remove inefficient naive-spec-version of shuffling algorithm now that protolambda's can be tested directly
2019-03-01 17:50:01 -06:00
Yuriy Glukhov 4c31f50889
Comments addressed 2019-03-01 22:09:20 +02:00
Yuriy Glukhov bed6510da3
Initial full sync impl 2019-03-01 19:39:38 +02:00
Zahary Karadjov 68a39a21be Mysterious fix for the mysterious build issue 2019-03-01 18:51:37 +02:00
Dustin Brody 794f6aed89
fix https://github.com/status-im/nim-beacon-chain/issues/145 (#147) 2019-03-01 14:39:17 +00:00
Jacek Sieka ddb3a9cd94
Revert "Add some missing RLP serialization routines (#144)" (#146)
This reverts commit 859aa327cb.
2019-03-01 07:34:37 -06:00
zah 859aa327cb Add some missing RLP serialization routines (#144)
* Add some missing RLP serialization routines
2019-02-28 15:50:21 -06:00
Dustin Brody 1cfea240ed Complete scan through v0.3.0 spec (#141)
* Complete scan through v0.3.0 spec; remaining known discreptancy is in bls_verify_multiple

* switch remaining slot-misuses of get_active_validator_indices to epochs

* spec workaround

* finish first 0.3.0 pass by getting slow, naive version of bls_verify_multiple running
2019-02-28 15:24:43 -06:00
Jacek Sieka 125231d321
add initial block pool (#139)
* implement in-memory block graph
* store tail block in database
* resolve unknown parents by syncing them from peers
* introduce concept of resolved blocks and attestations - those that
follow minimal protocol rules
* update state head lazily
* log more stuff
* shortHash -> shortLog
* start 9/10 beacon nodes by default, last can be started manually
* see also #134
* fix start.sh epoch length
2019-02-28 15:21:29 -06:00
Mamy Ratsimbazafy 20f99db058 remove unused epoch template (#142) 2019-02-28 08:17:57 -06:00
Yuriy Glukhov 5b60c3b45f Enable discovery service regardless bootnodes are provided 2019-02-28 09:48:33 +02:00
Dustin Brody 34e6149f89
fix crosslink epoch refs to begin with GENESIS_EPOCH; more minor epoch processing tweaks/unstubbed-code (#137)
* fix crosslink epoch refs to begin with GENESIS_EPOCH; more minor epoch processing tweaks/unstubbed-code

* remove obsolete TODO and use humanEpochNum
2019-02-27 13:58:07 +00:00
Dustin Brody 0305432a87 implement and call process_slashings and process_exit_queue from processEpoch; update a few more epoch processing details to be consistent with 0.3.0; add rest of final upates per epoch (#136) 2019-02-26 10:54:08 +01:00
Dustin Brody 60e6a9fedc assert relationship between constants necessary to avoid underflow 2019-02-25 14:58:04 +00:00
Dustin Brody eda43b5b1c adjust copyright comment form 2019-02-25 14:58:04 +00:00
Dustin Brody 6dab867848 cleanup 2019-02-25 14:58:04 +00:00
Dustin Brody 8baab40ea6 much faster shuffling based on observations from https://github.com/protolambda/eth2-shuffle enough that it's not easy to get exact speedup numbers 2019-02-25 14:58:04 +00:00
Dustin Brody f77016af68 switch sense of bls_verify check 2019-02-25 14:58:04 +00:00
Dustin Brody de2cad0e12 switch from legacy pre-spec sum_effective_balances(...) to get_total_balance(...) 2019-02-25 14:58:04 +00:00
Dustin Brody a42601e29e initial epoch processing update switching from slot-oriented to epoch-oriented, using newer helper functions 2019-02-25 14:58:04 +00:00
Dustin Brody fb3bbfccaa small 0.3.0 adjustments; comment updates/clarifications; add Transfer processing 2019-02-25 14:58:04 +00:00
Yuriy Glukhov a62d25ab98
Merge pull request #131 from status-im/state-replay
initial state replay implementation
2019-02-22 18:04:48 +02:00
Dustin Brody 2d52d5cbfe convert some asserts to doAsserts to keep them in release mode builds; rename get_initial_beacon_state to get_genesis_beacon_state to track spec; switch target spec version to 0.3.0; switch references to penalize_validator to slash_validator/slashValidator to track spec; make some function returns safer by omitting 'return' (#132)
* convert some asserts to doAsserts to keep them in release mode builds; rename get_initial_beacon_state to get_genesis_beacon_state to track spec; switch target spec version to 0.3.0; switch references to penalize_validator to slash_validator/slashValidator to track spec; make some function returns safer by omitting 'return'

* 2x shuffling speedup by hoisting pivot calculations per https://github.com/protolambda/eth2-shuffle
2019-02-22 10:56:45 +01:00
Jacek Sieka 4c2d212781
updates
* comment on last finalized state
* use explicitly named proc's instead of type tricks
2019-02-21 15:38:26 -06:00
Jacek Sieka 1d9c91d230
cleanups
* add ancestor getter to db layer, and use
* remove head block call
* add trivial beacon chain db test
2019-02-21 11:20:50 -06:00
Jacek Sieka 2d307e2257
initial state replay implementation
* fix initial attestation pool on reordered attestations
* simplify db layer api
* load head block from database on startup, then load state
  * significantly changes database format
* move subscriptions to separate proc's
* implement block replay from historical state
* avoid rescheduling epoch actions on block receipt (why?)
* make sure genesis block is created and used
* relax initial state sim parameters a bit
2019-02-20 22:42:17 -06:00
Dustin Brody 1d13007627 More spec v0.3.0 notations and updates (#129)
* tighten get_active_index_root epoch assertion bound

* implement merkle_root

* rm process_penalties_and_exits; add slash_validator

* rm rm'd and unused constant
2019-02-20 14:35:27 -06:00
Dustin Brody 132f1147a2 spec 0.3.0 updates: EPOCH_LENGTH -> SLOTS_PER_EPOCH, SlotNumber -> Slot, EpochNumber -> Epoch (#127)
* spec 0.3.0 updates: EPOCH_LENGTH -> SLOTS_PER_EPOCH, SlotNumber -> Slot, EpochNumber -> Epoch
2019-02-19 19:33:58 -06:00
Jacek Sieka 4670d6c98a naive attestation pool, in preparation of fork choice integration (#125)
* move attestation pool to separate file
* combine attestations lazily when needed
* advance state when there's a gap while attesting
* compile beacon node with optimizations - it's tooo slow right now
* log when unable to keep up
2019-02-20 00:35:02 +01:00
Dustin Brody 1269e001ad get most 0.3.0 spec renamings (#126)
* get most 0.3.0 spec renamings
2019-02-19 17:07:56 -06:00
Jacek Sieka c64cd2f275 a few simple SlotNumber cleanups 2019-02-18 22:43:33 +02:00
Mamy Ratsimbazafy 8e48bac58b
[WIP] Fix some issues after introducing "humaneEpochNumber" (#118)
* Fix some issues after introducing "humaneEpochNumber" + use slot_duration to 4

* Address humane comments

* Fix comment
2019-02-18 16:58:34 +01:00
Jacek Sieka 0e9cc20cfe beacon node simplifications (#116)
* remove previous randao scheme code
* fix some epoch logging
* move genesis time selection to state generation
2019-02-15 17:33:32 +01:00
Dustin Brody d3f97c1d5c
Merge pull request #115 from status-im/nus
complete switching to v0.2.0 RANDAO scheme, with verification enabled
2019-02-15 10:57:14 +00:00
Dustin Brody 6bef36d418 remove more old RANDAO vestiges 2019-02-14 14:03:45 -08:00
Dustin Brody 8fbf87aa8c complete switching to v0.2.0 RANDAO scheme, with verification enabled 2019-02-14 13:41:04 -08:00
Jacek Sieka a71424d768
add more config options and log compile parameters (#114)
default sim runs with smaller chain now
2019-02-14 13:32:33 -06:00
Mamy Ratsimbazafy c2a52d7fc5 Allow faster slots (#112)
* Allow compile time configurable slot time
2019-02-14 13:15:10 +01:00
Dustin Brody 55ac8d337b some v0.2.0 updates and fixes a bug that should enable simulation to continue longer (#110)
* some v0.2.0 updates and fixes a bug that should enable simulation to continue longer
2019-02-14 04:16:50 +01:00
Dustin Brody 31a4c7255f more v0.2.0 spec updates (#107)
* switch to v0.2.0 shuffling scheme

* fix some >80 char lines; add get_total_balance helper function (used in epoch processing, etc); note intended removal of repeat_hash and users thereof; add some int_to_bytes and bytes_to_int helpers; more underflow comments; add Gwei type; adjust generate_seed to include additional entropy source; note egregious inefficiencies

* minor code cleanup

* move some general helper functions to helpers.nim and clarify little-endian comments
2019-02-13 11:26:32 +01:00
Mamy Ratsimbazafy b60fa131f1
Fix simulation regression due to custody_bitfield (#108) 2019-02-12 23:50:02 +01:00
Mamy Ratsimbazafy 899c8e5df9
Bitfield debugging asserts (#106)
* debugging asserts

* Ensure initialization of custody_bitfield for mock Attestations
2019-02-12 18:07:44 +01:00
Yuriy Glukhov 56a48cc60f Fixed epoch actions (#105)
* Fixed epoch actions

* debugging asserts

* Revert "debugging asserts"

This reverts commit 9a11f5a7d8.
2019-02-12 16:56:58 +01:00
Dustin Brody 433e2f78c2
Merge pull request #104 from status-im/oug
Spec updates for 0.2.0
2019-02-11 15:48:00 +00:00
Dustin Brody f1cbc4a9dc adjust indentation 2019-02-11 16:32:22 +01:00
Mamy Ratsimbazafy f7feb9a6c9
Delay validators and add more log in crosslink asserts (#103) 2019-02-11 16:29:21 +01:00
Dustin Brody b03ac3427f split out get_previous_epoch into helper function 2019-02-11 16:10:46 +01:00
Dustin Brody d44999abc5 latest_crosslink_root -> latest_crosslink; adjust checkAttestation; add SHUFFLE_ROUND_COUNT and FORK_CHOICE_BALANCE_INCREMENT constants; some additional casts because a few places where ValidatorIndex was used are now uint64; remove some old commented-out lines of code from shardcommittee transition 2019-02-11 15:32:27 +01:00
Dustin Brody 4747477160 more checking that various functions match v0.1; some variable/etc renaming; fix is_double_vote to check epoch, not slot equivalence; finish verify_slashable_attestation; adjust is_surround_vote logic (#100) 2019-02-08 00:07:15 +01:00
Zahary Karadjov c567bc410f
Use human-readable names for the slot numbers 2019-02-07 22:14:08 +01:00
Yuriy Glukhov f1f1beb703 Fixed block proposal signature domain 2019-02-07 21:55:44 +01:00
mratsim 1e574cf75a Hacky (?) fix #97 assert fails 2019-02-07 21:22:46 +01:00
Dustin Brody 072c1607fd implement new RANDAO reveals (#92)
* implement new RANDAO reveals
2019-02-07 21:13:10 +01:00
Dustin Brody f2545318db get_crosslink_committees_at_slot updates (#95) 2019-02-07 20:27:13 +01:00
Dustin Brody 2794181889 Spec updates (#94)
* s/slot_included/inclusion_slot/; s/participation_bitfield/aggregation_bitfield/; rearrange some type definitions to match spec order

* a couple more Eth1Data-related spec syncs
2019-02-07 17:55:48 +01:00
Zahary Karadjov 31abd31d8a Fix an int conversion issue 2019-02-07 17:53:02 +01:00
Zahary Karadjov dfcd22519b
Fix for getting the current epoch number 2019-02-07 16:40:00 +01:00
Dustin Brody bc44744acd rm unused work_pool module (#90)
* rm unused work_pool module

* preserve comment from Danny about signature aggregation
2019-02-07 16:28:59 +01:00
Jacek Sieka ed8f7ab475
schedule epoch actions for all 64 slots in an epoch 2019-02-07 13:18:00 +01:00
Jacek Sieka 976e1e1f10
spec: remove unused code, fix state_sim speed 2019-02-07 13:03:02 +01:00
cheatfate 20e7ca48b5 Fix `validator not in registry` and `$` bugs. 2019-02-07 12:51:21 +02:00
Dustin Brody 2d71b998c7 Spec update to v0.1 (#88)
* update based on https://github.com/ethereum/eth2.0-specs/pull/483
* more spec updates
2019-02-07 11:43:21 +01:00
Dustin Brody 788b093284
Merge pull request #87 from status-im/vdy
Spec updates
2019-02-06 22:52:36 +00:00
Dustin Brody 74cb86ec21 more spec changes 2019-02-06 21:37:21 +01:00