Commit Graph

135 Commits

Author SHA1 Message Date
Zahary Karadjov 82282928c3 Allow setting the number of missing nodes 2019-03-29 11:32:25 +02:00
Zahary Karadjov c32a833bf7 Support for running a local simulation with arbitrary number of validators 2019-03-29 11:32:25 +02:00
Jacek Sieka 2f86bf21c8
attestation test: fix compile & disable temporarily 2019-03-28 13:15:09 -06:00
Ștefan Talpalaru f8fbe0ff3b Makefile that only works in a Git submodule (#210)
- updated README.md
- beacon_chain.nimble: accept compilation flags as params
- nim.cfg: enable `--opt:speed` at the top level
- simulation scripts: allow overriding GIT_ROOT and do some refactoring
2019-03-28 09:18:59 -06:00
Zahary Karadjov 5f4581240e Aha, the logging issue depended on options set in start.sh; Worked-around 2019-03-28 16:54:50 +02:00
Jacek Sieka f9e0418b5b fetch multiple blocks at a time
* avoid crash on invalid block production (fixes #209)
* fetch blocks every second, roughly
* fix start.sh param order
* run beacon node sim at slower pace
2019-03-28 10:42:16 +02:00
Jacek Sieka 3ad6053298 Use block, slot tuple to idenfiy fork
this is the beginning of tracking block-slots more precisely, so we can
the justification epoch slot bug.

* avoid asyncDiscard (swallows assertions)
* fix attestation delay
* fix several state root cache bugs
* introduce workaround for genesis epoch spec issue
2019-03-28 10:32:09 +02:00
Zahary Karadjov b3d23c052c Switch to 30 seconds slot duration; UX improvements
* On the testnet site, the first N instead of the last N validators
  will be considered reserved for users

* The number of validators is consistently named `totalValidators`
  in all commands

* Proper keys are generated by default in validator_keygen
2019-03-27 18:13:39 +02:00
Jacek Sieka 81c66fbce0
use signed_root as canonical block root (#211)
* no need to pass prev_block_root when updating state
* some Slot fixes
* fix `hash_tree_root` for Slot, Epoch
* detect missing hash_tree_root type support
* remove some <0.5 state checks
2019-03-26 19:32:35 -06:00
Zahary Karadjov 803a0570be deleted an accidentally commited submodule 2019-03-26 18:42:02 +02:00
Mamy André-Ratsimbazafy 808c5edfc0 Add more fork choice logs 2019-03-26 17:35:09 +02:00
Jacek Sieka 1b0e67c88c
ssz: update to 0.5.1:ish (#202)
* ssz: update to 0.5.1:ish
* slightly fewer seq allocations
* still a lot of potential for optimization
* fixes #174
* ssz: avoid reallocating leaves (logN merkle impl)
2019-03-25 10:46:31 -06:00
Dustin Brody 9f55b4646c
More 0.5.1 spec updates (#195)
* rm gone-in-0.5.0 Proposal, verifyBlockSignature, and slot check which moved to spec function processBlockHeader

* mark get_attestation_participants and get_epoch_committee_count as 0.5.1; finish updating processAttestations to 0.5.1; add kludgy workaround for bug relating to get_winning_roots_etc using crosslink_data_root as index when we have that as ZERO_HASH for all leading to it confusing attesters on different shards; rm BeaconState.latest_attestations, which splits into previous_epoch_attestations and current_epoch_attestations

* Fix CI due to removed latest_attestations field
2019-03-22 18:33:12 +00:00
Zahary Karadjov 0a027e410a Address review concerns and make some steps towards json logging 2019-03-22 17:04:17 +02:00
Jacek Sieka ee89ef1c79
cleanups (#189)
* add simple bitfield type (fixes #19)
* fix bit endianess to match spec
* consolidate attestation and block logging
* cruft cleanup
* run optimizer on tests, speeds up build
2019-03-20 14:01:48 -06:00
Dustin Brody 13c7f7fcec begin refactoring 0.5.0 epoch transition (#185)
* begin refactoring 0.5.0 epoch transition at justification and finalization

* update processTransfers to 0.5.0 spec

* update get_crosslink_committees_at_slot to 0.5.0 except for noted workaround for pre-0.5.0 spec bug related to epoch boundary conditions, to be flipped in more coordinated way with other epoch processing changes

* mark ejection processing as 0.5.0

* 0.5.0 epoch update finalization

* rm BEACON_CHAIN_SHARD_NUMBER to complete updating miscellaneous constants to 0.5.0

* mark verify_slashable_attestation as 0.5.0

* update process_attester_slashing to 0.5.0

* update process_slashings(...) to 0.5.0 and mark process_exit_queue as 0.5.0

* refactor epoch processing by implementing 0.5.0's update_registry_and_shuffling_data(...)
2019-03-19 20:19:46 -06:00
Zahary Karadjov 9930fa5bef Fix a wrong bootstrap port 2019-03-20 03:34:04 +02:00
Jacek Sieka 71670ca9c9
make nat setting optional when running sim 2019-03-19 18:12:54 -06:00
Zahary Karadjov 6a35d3584d Automate the creation of the network metadata files
With these changes, running a simulation is very close to running
an actual testnet. Some checks have been added in the client to
make sure you are not connecting to an incompatible network (e.g.
a network running with a different number of shards).
2019-03-19 20:27:54 +02:00
Jacek Sieka d7f972a47e allow setting advertised external ip
* detect it in start.sh
2019-03-19 10:50:09 +02:00
Dustin Brody 2bc632c29b
AttestationData.justified epoch -> AttestationData.source_epoch and AttestationData.latest_crosslink -> AttestationData.previous_crosslink; implement verify_merkle_branch and update process_deposit to use it; rm BeaconState.batched_block_roots; Transfer.{from, to} -> Transfer.{sender, recipient}; update Fork data type and code to use bytes4; implement int_to_bytes8 (close to metaprogramming time); adjust signed_root and all callers to use 0.5.0 calling signature; processDepositRoot -> processEth1Data (#182) 2019-03-18 15:42:42 +00:00
Zahary Karadjov 6bb38095c9 Command-line and data storage handling for testnets 2019-03-18 12:45:29 +02:00
Dustin Brody 798197879b
Revert "AttestationData.justified epoch -> AttestationData.source_epoch and AttestationData.latest_crosslink -> AttestationData.previous_crosslink; implement verify_merkle_branch and update process_deposit to use it; rm BeaconState.batched_block_roots; Transfer.{from, to} -> Transfer.{sender, recipient}; update Fork data type and code to use bytes4; implement int_to_bytes8 (close to metaprogramming time); adjust signed_root and all callers to use 0.5.0 calling signature; processDepositRoot -> processEth1Data"
This reverts commit 18f1d000a1.
2019-03-17 15:24:50 -07:00
Dustin Brody 18f1d000a1
AttestationData.justified epoch -> AttestationData.source_epoch and AttestationData.latest_crosslink -> AttestationData.previous_crosslink; implement verify_merkle_branch and update process_deposit to use it; rm BeaconState.batched_block_roots; Transfer.{from, to} -> Transfer.{sender, recipient}; update Fork data type and code to use bytes4; implement int_to_bytes8 (close to metaprogramming time); adjust signed_root and all callers to use 0.5.0 calling signature; processDepositRoot -> processEth1Data 2019-03-17 15:17:40 -07:00
Dustin Brody f36c2d86dc begin 0.5.0 spec update (#179)
* begin 0.5.0 spec update: parent_root -> previous_block_root, BeaconState.justified_epoch -> BeaconState.current_justified_epoch, DOMAIN_PROPOSAL -> DOMAIN_BEACON_BLOCK, temporarily rename BeaconBlockHeader to BeaconBlockHeaderRLP to allow for gradual re-merging without disrupting RLP; mark a few unchanged functions and data types, implement get_temporary_block_header/get_empty_block/should_update_validator_registry/processBlockHeader/cacheState; update a few others

* a dozen or so more trivial mostly comment changes, finding more unchanged parts of 0.5.0

* several more trivial changes; goal is to reduce noise around the upcoming substantial changes (epoch processing, etc)
2019-03-16 13:52:37 -06:00
Ștefan Talpalaru 9ff1eb4ac8
Merge branch 'master' of github.com:status-im/nim-beacon-chain 2019-03-14 00:05:12 +01:00
Ștefan Talpalaru de295619be
assert() -> doAssert() 2019-03-14 00:04:43 +01:00
Jacek Sieka b0f4034060
implement fork choice (#175)
* keep track of a finalized block
* keep track of all justified blocks
* use naive spec version of LMD ghost
* cache slot number and a few more things in BlockRef
* keep track of the latest vote of each validator
* depend less on the state of node.state (it's a cache, effectively)
2019-03-13 16:59:20 -06:00
Bruno Škvorc e69702bab5 Use environment variables (#176)
Passing along no-name command line arguments when using make (from Nimbus) would need make target revamps that are a bit clumsy AFAIK, so using environment variables in the spirit of USE_MULTITAIL to get the number of validators and nodes makes more sense to me.

This lets us run the sim from nim-beacon-chain with

```bash
VALIDATORS=512 NODES=100 USE_MULTITAIL="yes" tests/simulation/start.sh
```

and from Nimbus with:

```bash
VALIDATORS=512 NODES=100 USE_MULTITAIL="yes" make eth2_network_simulation
```

Also makes for a more verbose launch process.
2019-03-13 06:44:30 -06:00
Dustin Brody 1479bae22f implement distinct Epoch type and continue 0.4.0 spec update (#173)
* implement distinct Epoch type
* update two more spec 0.4.0 markers, leaving just deposit processing
2019-03-12 19:46:44 -06:00
Dustin Brody 50a0948bc0 Mostly convert to using distinct Slot type (#172) 2019-03-12 16:21:32 -06:00
Mamy André-Ratsimbazafy 6dea18c140
Squashed commit of the following:
commit bed6909f33
Author: Mamy André-Ratsimbazafy <mamy_github@numforge.co>
Date:   Mon Mar 11 12:22:35 2019 +0100

    Cleanup imports

commit 241f62ed8d
Author: Mamy André-Ratsimbazafy <mamy_github@numforge.co>
Date:   Mon Mar 11 12:06:00 2019 +0100

    Group beacon types to prevent recursive imports
2019-03-12 16:03:14 +01:00
Zahary Karadjov 903cb8a8b5 A LibP2P-based networking stack;
To enable it, comment out the 'withLibp2p' line in nim.cfg

The history was squashed in order to remove an accidentally
commited binary file.

Other changes:

* SSZ was adapted to use the common serialization framework

* gossibsup.subscribe is not using async handlers at the moment
  and this allowed me to simplify it
2019-03-12 13:36:55 +02:00
Yuriy Glukhov b587455e7b Use BlockPool for slot->blocks mapping (#166) 2019-03-11 09:38:36 -06:00
Dustin Brody 13de015038 more 0.4.0 spec updates: renamings, continue Proposal updates, and note several more functions which were verified not to have changed in 0.4.0 (#163) 2019-03-11 09:33:24 -06:00
Jacek Sieka aec31b1126 Updates for finalizing chain! (#162)
* set epoch_boundary_root - chain finalizes!
* fix slotStart to offset GENESIS_SLOT
* work around bug that will be fixed by
https://github.com/ethereum/eth2.0-specs/pull/732
* compile with debug info (there was a GC-related crash in C land)
2019-03-09 04:23:14 +00:00
Dustin Brody 1afdcda62d
spec version 0.4.0 update: shard_block_root -> crosslink_data_root; switch from validator.slashed_epoch -> validator.slashed approach; a couple assert -> doAssert; note fix for underflow-prone spec changes in checkAttestation; GENESIS_SLOT change from 2^63 to 2^32; refactor Proposer and ProposalSlashing structure; add signed_root; remove duplicate slashValidator (#159)
* spec version 0.4.0 update: shard_block_root -> crosslink_data_root; switch from validator.slashed_epoch -> validator.slashed approach; a couple assert -> doAssert; note fix for underflow-prone spec changes in checkAttestation; GENESIS_SLOT change from 2^63 to 2^32; refactor Proposer and ProposalSlashing structure; add signed_root; remove duplicate slashValidator

* re-apply shard_block_root -> crosslink_data_root

* remove incorrect humaneSlotNum

* add (run-time only, alas) sanity check on signed_root
2019-03-08 17:44:31 +00:00
Jacek Sieka 6bcefc0e42
verify blocks before storing in block pool / database (#158)
* fix state db lookup typo
* fix randao reveal slot when proposing blocks
* only store blocks that can be applied to a state
* store state at every epoch boundary (yes, needs pruning!)
* split out state advancement function when there's no block
* default state sim to 0.9 attestation ratio
2019-03-08 10:40:17 -06:00
Jacek Sieka 4d55cf8eea beacon node sim: various improvements (fixes #111) (#156)
* allow running more or fewer validators
* use deterministic key generation for tests to avoid exhausting system
RNG
* update README with simulator docs
* write the data of each validator to separate file, instead of a big
chainstart.json (makes it easier to run different validator counts)
2019-03-07 13:59:28 +00:00
Dustin Brody b8402e9809
speed up epoch processing by caching shuffling results (#153)
* speed up epoch processing by caching shuffling results
2019-03-05 19:10:36 +00:00
Dustin Brody 81d5becc7b Switch hash functions, prevent underflow & verify that shuffling works (#149)
* prevent balance underflow, per spec implementation note

* verify that permutation shuffing works: add YAML dependency to be used for ingesting test cases; switch from blake2 to keccak256 to match EF test cases; remove inefficient naive-spec-version of shuffling algorithm now that protolambda's can be tested directly
2019-03-01 17:50:01 -06:00
Dustin Brody 1cfea240ed Complete scan through v0.3.0 spec (#141)
* Complete scan through v0.3.0 spec; remaining known discreptancy is in bls_verify_multiple

* switch remaining slot-misuses of get_active_validator_indices to epochs

* spec workaround

* finish first 0.3.0 pass by getting slow, naive version of bls_verify_multiple running
2019-02-28 15:24:43 -06:00
Jacek Sieka 125231d321
add initial block pool (#139)
* implement in-memory block graph
* store tail block in database
* resolve unknown parents by syncing them from peers
* introduce concept of resolved blocks and attestations - those that
follow minimal protocol rules
* update state head lazily
* log more stuff
* shortHash -> shortLog
* start 9/10 beacon nodes by default, last can be started manually
* see also #134
* fix start.sh epoch length
2019-02-28 15:21:29 -06:00
Ștefan Talpalaru 8f64ab6add start.sh: use native Windows paths on Mingw 2019-02-28 09:50:17 +02:00
Dustin Brody 34e6149f89
fix crosslink epoch refs to begin with GENESIS_EPOCH; more minor epoch processing tweaks/unstubbed-code (#137)
* fix crosslink epoch refs to begin with GENESIS_EPOCH; more minor epoch processing tweaks/unstubbed-code

* remove obsolete TODO and use humanEpochNum
2019-02-27 13:58:07 +00:00
Dustin Brody 8baab40ea6 much faster shuffling based on observations from https://github.com/protolambda/eth2-shuffle enough that it's not easy to get exact speedup numbers 2019-02-25 14:58:04 +00:00
Yuriy Glukhov a62d25ab98
Merge pull request #131 from status-im/state-replay
initial state replay implementation
2019-02-22 18:04:48 +02:00
Dustin Brody 2d52d5cbfe convert some asserts to doAsserts to keep them in release mode builds; rename get_initial_beacon_state to get_genesis_beacon_state to track spec; switch target spec version to 0.3.0; switch references to penalize_validator to slash_validator/slashValidator to track spec; make some function returns safer by omitting 'return' (#132)
* convert some asserts to doAsserts to keep them in release mode builds; rename get_initial_beacon_state to get_genesis_beacon_state to track spec; switch target spec version to 0.3.0; switch references to penalize_validator to slash_validator/slashValidator to track spec; make some function returns safer by omitting 'return'

* 2x shuffling speedup by hoisting pivot calculations per https://github.com/protolambda/eth2-shuffle
2019-02-22 10:56:45 +01:00
Jacek Sieka 4c2d212781
updates
* comment on last finalized state
* use explicitly named proc's instead of type tricks
2019-02-21 15:38:26 -06:00
Jacek Sieka 1d9c91d230
cleanups
* add ancestor getter to db layer, and use
* remove head block call
* add trivial beacon chain db test
2019-02-21 11:20:50 -06:00