Commit Graph

548 Commits

Author SHA1 Message Date
Zahary Karadjov 806836714a Add an accidentally forgotten fix 2019-10-01 17:50:13 +03:00
Eugene Kabanov 13bde5aee3 Add gossipsub messages monitoring tool. (#458)
* Add gossipsub messages monitoring tool.

* Add support of bootstrap nodes file.

* Move topic constants to spec/network.nim.

* Add ability to monitor custom topics.
2019-10-01 15:52:28 +02:00
Mamy Ratsimbazafy 48d22d53dc
Ensure that we use the interop deposit signatures (#467)
* Ensure that we use the interop deposit signatures

* Add all interop keys to the test
2019-10-01 15:44:38 +02:00
Jacek Sieka 66f03f491f
log raw incoming gossip data 2019-09-28 21:20:09 +02:00
Zahary Karadjov 66984c44ce Another innocent fix from the interop branch 2019-09-27 23:29:58 +03:00
Zahary Karadjov a336285922 Fix a potential OOB error in SSZ deserialization 2019-09-27 23:29:58 +03:00
Zahary Karadjov c39d4f1817 Fix an overflow in the justification bits processing 2019-09-27 23:29:58 +03:00
Zahary Karadjov 1555efd9d8
Implement the latest modification of the spec 2019-09-27 19:10:37 +03:00
Zahary Karadjov 37043f0d91
Handle gracefully a pre-mature closing of a libp2p stream by another peer 2019-09-27 19:10:24 +03:00
Zahary Karadjov b5fad0c9e8
Add more diagnostic for crashes during SSZ serialization 2019-09-27 19:07:22 +03:00
Zahary Karadjov 2bbe2cb23c
Implement the outputBootstrapNodes option (taken from the interop branch) 2019-09-27 19:05:17 +03:00
Zahary Karadjov b56c96a4fd
Support downloading SSZ genesis files with curl 2019-09-27 18:59:28 +03:00
Zahary Karadjov 5152c42c8e
Don't crash hard on P2P daemon errors during the initial connection 2019-09-27 18:58:51 +03:00
Zahary Karadjov 0eaa433e84
Automated scripts for resetting the Status testnets
For detailed instructions, please see

https://github.com/status-im/nimbus-private/blob/master/testnets-maintenance.md
2019-09-26 18:58:51 +03:00
Mamy André-Ratsimbazafy 90b20fab52 Domain is now array[8, byte] instead of uint64 2019-09-26 17:53:59 +03:00
Mamy Ratsimbazafy a7a1f78499
Fix TRACE chronicles_log_level (#451) 2019-09-23 17:00:10 +02:00
Mamy Ratsimbazafy b100ceef56
Add detailed finalization and justification debug logs (#449)
* Add detailed finalization and justification debug logs

* log was wrongfully reporting rule 123 instead of rule 12
2019-09-23 15:48:26 +02:00
Mamy Ratsimbazafy 0b68f3dc63
[Interop branch] Log improvements - part 1 (#439)
* use service/category/process for blockpool logs
Only track fork choice logs in block pool (vs beacon_node)
Reduce verbosity on usual event in block pool

* rework beacon node logs

* log for attestations in blockpool

* log - att pool improvement

* use logScope and topics cf review and discussion

* use 7 letters for beacon_node

[log] report peers at slot start + fix bracket prefix [Block pool] Attestation sent
2019-09-23 11:35:41 +02:00
Jacek Sieka 42a469603d
simplify startup, describe a few startup tricks 2019-09-11 20:29:57 -04:00
Jacek Sieka 65ff8f2886
remove networkId (unused) 2019-09-11 20:29:57 -04:00
Jacek Sieka ab2cce3951
add quickStart option that avoids network metadata 2019-09-11 20:29:57 -04:00
mratsim c619b9557a
Validator pool update to use BLS domain 2019-09-11 20:29:57 -04:00
Zahary Karadjov 2aee749ead
Save a genesis SSZ file 2019-09-11 20:29:57 -04:00
Mamy Ratsimbazafy 988a13b872 Official operations - deposits unit test (#408)
* [Test] Official operations - deposits unit test

* Allow ignoring deposits with invalid signature

* We need stacktraces to debug windows 64 issue https://github.com/status-im/nim-beacon-chain/pull/408#issuecomment-529236359

* fix naming of unit test proc

* Revert "We need stacktraces to debug windows 64 issue https://github.com/status-im/nim-beacon-chain/pull/408#issuecomment-529236359"

This reverts commit 04b8b05162.

* skip windows-64 in CI #435

* proposer slashing started to crash as well on win-64 #435
2019-09-11 21:10:54 +00:00
Mamy Ratsimbazafy 3dc2b87e6a Transfer - split process_transfer/processTransfers + tests + fixes (#422)
* Prepare test suite for transfers

* split API process_transfer / processTransfers

* Add range checks on transfer

* Fix invalid transfer conditions

* don't test on windows 64-bit #435
2019-09-11 20:29:00 +00:00
Mamy Ratsimbazafy 8676bbf388 Add proposer slashing tests (#431)
* Add proposer slashing tests

* typo in import
2019-09-11 07:50:07 +00:00
Mamy Ratsimbazafy 6be2e24a8f
Attester slashing - split process_attester_slashing/processAttesterSlashings + tests (#424)
* attester slashing tests - pending #415

* split process_attester_slashing/processAttesterSlashings

* Add logs to attester_slashing

* deactivate bls tests for now (https://github.com/status-im/nim-beacon-chain/issues/429) and cherry-pick from 60f2437dd2
2019-09-10 18:03:06 -04:00
Dustin Brody f72de67f63 Proposer slashing is 0.8.3 (#425)
* minimal refactor of proposer slashings to 0.8.3; no semantic difference

* actually mark spec
2019-09-10 11:29:46 -04:00
Zahary Karadjov f583135a39
The --stateSnapshot option now accepts ssz files as well 2019-09-10 10:13:49 -04:00
Jacek Sieka e470bbb52e
don't sign libp2p messages 2019-09-10 10:13:02 -04:00
Zahary Karadjov c3adac44f2
Use Secp256k1 keys in accordance to the interop spec 2019-09-10 10:11:56 -04:00
Zahary Karadjov cdff79ec6d More sync fixes
* Fix IncompleteData issues brought by the new spec-compliant stream closing
* Fix logic errors in the sync algorithms
2019-09-10 09:49:54 -04:00
Zahary Karadjov 886b92319f Implement the response size limits 2019-09-10 09:49:54 -04:00
Jacek Sieka 86fc9a1f9e fix nil access to mainchainMonitor 2019-09-10 09:49:54 -04:00
Zahary Karadjov a83aa83644 Working BeaconSync
Changes:

* Do not send separate network packets for response codes and msg
  len prefixes

* Close streams according to the spec

* Implement more timeouts according to the spec

* Make hello requests during syncing to update our knowledge of
  the head block of the other peer.
2019-09-10 09:49:54 -04:00
Zahary Karadjov 4a54fb4103 Cleaned up obsolete BeaconSync code; Added some open questions regarding fetchAncestorBlocks 2019-09-10 09:49:54 -04:00
Zahary Karadjov 9dec05f9c9 Sending chunked responses 2019-09-10 09:49:54 -04:00
Zahary Karadjov b120a60493 Reading chunked responses 2019-09-10 09:49:54 -04:00
Zahary Karadjov 2bbfa8c877 Changes related to the new ETH interop spec
* Hello is no longer a handshake message
  (all handshakes related code was deleted for clarity)

* Deal with the single-parameter inlining defined in the new spec
2019-09-10 09:49:54 -04:00
Mamy Ratsimbazafy 81b47f35d1 Voluntary exit - split single + multiple exits proc & tests (#421)
* add test suite for voluntary exit

* update API to process_voluntary_exit

* Add range check of validator_index for voluntary exits

* Revert to dual single + multiple voluntary exits API + enable in test suite

* no cache or mocking needed
2019-09-10 00:14:03 +00:00
Mamy Ratsimbazafy 061b6e0ddf
Test block headers (#418)
* Block headers tests

* Skip invalid_parent_root test due to https://github.com/status-im/nim-beacon-chain/issues/407

* Fix bls_verify with invalid sig. Activate block header tests
2019-09-09 14:40:59 -04:00
Yuriy Glukhov d4c480ff5f
Eth1 validators (#393) 2019-09-09 18:59:02 +03:00
Dustin Brody a4b4c89539 remove incorrect exit condition in get_attestation_details(...) (#416)
* remove incorrect exit condition in get_attestation_details(...)

* enable historical batch unit test, which now passes
2019-09-09 09:10:03 -04:00
Mamy Ratsimbazafy 3f446e6383
Attestations EF tests + Proper Real/Opaque BlsValue deserialization (#410)
* Add attestation unit test

* process_attestation doesn't throw exceptions

* Allow SSZ deserialization of both real and invalid signatures

* Add new process_attestation checks - pass all process_attestation tests

* Add sanity check for #361

* Fix SSZ testing after fromBytes/fromSSZBytes changes
2019-09-08 23:33:24 -04:00
Jacek Sieka 5f58af2112
fix ssz list, protocol names (#405) 2019-09-08 16:54:31 +02:00
Eugene Kabanov bb0e9de09d [WIP] Sync which using latest network specification. (#390)
* Fixed getBeaconBlocks() and getRecentBeaconBlocks() to use BlockPool, not db.

* Got the sync_protocol to compiling state; Removed all obsolete RPC calls
2019-09-08 15:08:44 +02:00
Mamy Ratsimbazafy 82b9e008d6
Fix genesis blockheader zero signature (supercedes #395) (#400) 2019-09-07 20:56:24 -04:00
zah 93cdb43f1e Integrate nim-metrics and add some metrics from the ETH2 spec (#394) 2019-09-07 19:48:05 +02:00
Dustin Brody 0c174036a5 update get_attesting_indices(...) to 0.8.3; mark IndexedAttestation, Attestation, PendingAttestation, and get_randao_mix(...) as 0.8.3; rm duplicate/dead code get_unslashed_attesting_indices(...) (#391) 2019-09-06 21:58:38 +02:00
Yuriy Glukhov e3bd4410d8
Revert "Eth1 validators" (#392)
This reverts commit bb8955bdbc.
2019-09-06 13:00:56 +03:00
Yuriy Glukhov bb8955bdbc Eth1 validators 2019-09-06 02:38:38 +02:00
Dustin Brody f882c92a42 fix network sim (#389)
* fix network sim

* mark BeaconState, state list/vector lengths, misc values, get_base_reward(...), verifyStateRoot(...), and process_slot(...) as 0.8.3; update minimal/mainnet config initial values to 0.8.3 by removing GENESIS_FORK_VERSION
2019-09-05 21:52:34 +02:00
Jacek Sieka 31db5d3a62 Deserialize serialized default bls values (#388)
Ugly workaround to quick-fix broken network/interop sim
2019-09-05 14:27:28 +00:00
zah 7ebf685fa3 Work-around for #373 (#385) 2019-09-05 10:27:26 +00:00
Mamy Ratsimbazafy ad240953ed Fix #378, int to bytes32, LATEST_RANDAO_MIXES, get_seed, get_crosslink_committee, get_compact_committee_root (#380) 2019-09-04 13:57:18 +00:00
Dustin Brody b9265e15f1
remove some superfluous/pointless spec version markers (#377) 2019-09-04 09:25:27 +00:00
Mamy Ratsimbazafy c11f37e550 V0.8.3 tests part 3 - sanity checks for slots and blocks (#375)
* Add sanity check for slot processing (also impacted by https://github.com/status-im/nim-beacon-chain/issues/373)

* use reportDiff also for all state tests vs EF

* initial sanity checks for blocks - workaround zero signature in block headers: https://github.com/status-im/nim-beacon-chain/issues/374

* Remove generic object variant compare commented code

* Add the one block state transition sanity checks

* generalize blocks test to multiple blocks

* simplify slots test runner

* Add official epoch transitions, sanity blocks, sanity slots to the test suite

* Fix index out-of-bounds in initiate_validator_exit - enable proposer slashings unittest
2019-09-03 18:02:21 +00:00
Mamy Ratsimbazafy 297e9079d4 V0.8.3 tests part2 - BLS + Shuffling + State transition epoch (#372)
* Update BLS fixtures to 0.8.3

* Bump fixtures with shuffling with minimal preset

* Update shuffling tests

* parseTest generic over file format (Json or SSZ)

* Initial crosslink parsing commit for debugging Nim crash

* Workaround https://github.com/status-im/nim-beacon-chain/issues/369

* Crosslink test works for minimal - https://github.com/status-im/nim-beacon-chain/issues/369 is back on mainnet

* Use ref objects to workaround https://github.com/status-im/nim-beacon-chain/issues/369

* Generalize state transition epoch test to all epoch tests

* Fix slashing (potential uint64 overflow in previous spec)

* Add a state debugging macro to deeply inspect the wrong fields

* make reportDiff visible

* Improve the debug state macro for containers
2019-09-03 03:12:09 +00:00
Jacek Sieka 6d798c821f log json errors when loading deposits 2019-09-02 22:14:18 +03:00
Jacek Sieka 9cfbdb5e16 more interop fixes
* fix eth1 interop block hash
* update initialize_beacon_state_from_eth1 to 0.8.3
2019-09-02 22:14:18 +03:00
Jacek Sieka 7b73b40bab interop updates
* add interop launcher scripts
* stick validator_keygen into beacon_node
* fix lmd ghost slot number on missing block
* use mocked eth1data when producing blocks
* use bls public key method for withdrawal credentials
* fix deposit domain
* prefer lowercase for a bunch of toHex
* build simulation binary in data folder to avoid data types confusion
2019-09-02 22:14:18 +03:00
Mamy Ratsimbazafy ca4f29caca v0.8.1 tests refactor (#326)
* Introduce new mocking proc to replace:
- makeFakeValidatorPrivKey
- hackPrivKey
- getNextBeaconProposerIndex
- addBlock
- makeBlock

* Add comments on datastructure unsynced with the spec

* Add merkle tree constructor and initial mocking for deposits (missing merkle proofs)

* [Mock] Implement sparse merkle tree and merkle proof builder

* [Mocking] Genesis deposits

* Add compact_committees_roots init + mock genesis state

* [Tests] Add first deposit test using the new mocking procedures

* [Tests -deposits] add at and over 32 ETH deposit tests

* [Tests - deposits] Add test for validator top-up

* [Tests -deposits] Mention the TODO to test for invalid conditions

* [Tests] Add stub to test "is_valid_genesis_state"

* [Merkle proofs] Implement round-trip checks

* Deactivate roundtrips test

* SSZ - use EF convention for hash_tree_root / hashTreeRoot

* [Tests - Attestation] Attestation mocking + initial test

* Add mocking + 3 new tests for valid attestations + mention future invalid attestation tests

* Add crosslinks test (1 failing to attestations in block being duplicated in state transition)

* Single attestation crosslink test - workaround https://github.com/status-im/nim-beacon-chain/issues/361

* Add test for failed crosslink penalty

* Rebase fixes + add refactored tests to test suite

* justif-finalization helpers first batch

* Add 234 finalization tests

* Fix justif test, Rule I 234 finalization does not happen with sufficient support.
(Also unittest check template does not fail properly in some cases)

* Add tests for all finalization rules

* Properly delete nim-byteutils following c91727e7e5 (diff-7c3613dba5171cb6027c67835dd3b9d4)

* use digest helper for deposit root
2019-08-28 12:07:00 +00:00
Bruno Škvorc a7998f5c3e
Update datatypes.nim 2019-08-27 15:03:00 +02:00
Bruno Škvorc 5254ec0e7f
Add assert message
Tripped me up during build, error was unnecessarily unspecific.
2019-08-27 14:59:57 +02:00
Dustin Brody 8665efbbd1 bump beacon chain versions to 0.8.3, which has no relevant changes in this regard (#358) 2019-08-26 11:09:47 +02:00
Dustin Brody cc0429a643
slightly update/rename verify_merkle_branch(...) to 0.8.2 is_valid_merkle_branch(...); mark check_attestation(...), get_active_validator_indices(...), get_seed(...), and BeaconBlockBody as 0.8.2; update compute_domain(...) 0.8.2 by reversing field order (#356) 2019-08-26 07:57:09 +00:00
Mamy André-Ratsimbazafy ad55889707
Fix SSZ typo fieldCaseDisciminator (declared in nim-serialization) 2019-08-24 16:52:58 +02:00
Dustin Brody c47e3903be rm some less-useful debug and tracing noise (#355) 2019-08-23 11:39:23 +02:00
andri lim 5f7fb03cf2 fix SszChunkMerkelizer 32bit failure 2019-08-22 20:31:13 +03:00
Zahary Karadjov 912563648e
Fix the json build; Don't long the non-attesting indices in the testnets 2019-08-21 14:21:08 +03:00
Zahary Karadjov 180e8ed02b
Fix a build issue after rebasing 2019-08-20 16:42:15 +03:00
Zahary Karadjov f477eb6877
Add a `sszList` helper for writing tests involving SSZ lists 2019-08-20 16:34:36 +03:00
Zahary Karadjov 74f630eb28
Better start-up logic when the beacon node is launched before the genesis time 2019-08-20 16:34:33 +03:00
Jacek Sieka 78f7a4a505 restore network sim finalization
* relax attestation validation when attestation is incoming but make it
more strict when adding to block
* share attestation validation logic between attestation pool and state
transition
* remove a bunch of redundant logging
* fix potential underflow in attestation delay checking
* fix committee used for attestation shard selection when attesting
* fix attestation data construction
2019-08-19 21:05:54 +03:00
Jacek Sieka 225465578c fix startup time casts, log fixes 2019-08-19 21:05:54 +03:00
Zahary Karadjov f5db2dd7b3
Fix the build with the latest Chronos 2019-08-15 20:43:06 +02:00
Zahary Karadjov 2fca9465e7
Fix the build with RLPx 2019-08-15 20:43:05 +02:00
Jacek Sieka e639cd95c7
humaneXNum -> shortLog 2019-08-15 18:01:55 +02:00
Zahary Karadjov 4daeebbf7e
Fix the build when json logging is enabled 2019-08-14 17:38:39 +02:00
Dustin Brody 3e587d4667
Justification/finalization amelioration using testnet1 shard count and epoch length (#331)
* working justification, 1 node, 64 validators

* closer to tn1 params: 256 validators

* more debugging output; switch to minimum test case

* working justification and finalization in local network simulation

* fix currently incorrect state transition/attestation test assumption
2019-08-14 08:56:32 +00:00
Dustin Brody 6704749eb0 mark unchanged 0.8.1 as 0.8.2 spec aspects (#336) 2019-08-09 14:08:00 +02:00
Yuriy Glukhov e8ca67fc9a Updated web3 api usage 2019-08-07 15:48:02 +03:00
Zahary Karadjov a897ab9c70
Attempt to make the test suite green 2019-08-07 06:09:26 +03:00
Zahary Karadjov 5ce50b3aca
Implement the latest networking spec
https://github.com/ethereum/eth2.0-specs/pull/1328
2019-08-07 05:22:28 +03:00
Zahary Karadjov 398ea55801
Implement the latest SSZ specification and integrate the official SSZ test suite 2019-08-05 03:18:48 +03:00
Yuriy Glukhov 46c9e99754 Cleanup web3 api usage 2019-07-31 16:39:26 +03:00
Yuriy Glukhov 24cbc0710f Make web3 way of getting genesis optional 2019-07-31 16:39:26 +03:00
Yuriy Glukhov 68c4cbf078 Updated deposit contract 2019-07-31 16:39:26 +03:00
Yuriy Glukhov 958e4cd3b4 Getting genesis through web3 2019-07-31 16:39:26 +03:00
Mamy André-Ratsimbazafy 3e622644e7
Remove committed binaries 2019-07-21 17:23:05 +02:00
Zahary Karadjov 33dfc8b2d5
Fix the compilation when json logging is used 2019-07-16 13:40:24 +03:00
Dustin Brody b049dded88
summarize why previous epoch attestations can be recapped where they are 2019-07-16 11:16:04 +02:00
Dustin Brody f6fd089ae7
Add non-attesting-validators instrumentation in process_justification_and_finalization(...) (#323)
* update process_justifications_and_finalizations(...) to 0.8.1
2019-07-16 07:34:11 +00:00
Mamy Ratsimbazafy d1d5497233
v0.7.x-v0.8.1 state transition refactor part2 (#322)
* State sim compiles and run

* Works on libp2p testnet1

* Successfully update the state transition tests

* naive update of attestation pool (failing attstations can arrive in any order)

* Remove an official assert to allow attestation reordering
2019-07-15 23:10:40 +02:00
Dustin Brody 712038fe21
mark applicable 0.8.0-compliant functions and data structures as 0.8.1 and update a couple of other comments (#321) 2019-07-15 12:57:18 +00:00
Dustin Brody c92806b950
include both static and dynamic check for invariants to ensure that get_crosslink_committee(...) can access all committees by ensure that SHARD_COUNT >= SLOTS_PER_EPOCH (#318)
* include both static and dynamic check for invariants to ensure that get_crosslink_committee(...) can access all committees by ensure that SHARD_COUNT >= SLOTS_PER_EPOCH

* add state_sim to CI, to ensure that it also doesn't break SHARD_COUNT/SLOTS_PER_EPOCH assumptions, along with just in general not breaking
2019-07-13 15:58:01 +00:00
Ștefan Talpalaru 2768a3e89e
Ctrl+C handling and:
- added a "quitProc" for shutting down the p2pd process, after seeing it
  survive an unhandled exception (this required a global var for the
  DaemonApi instance)
- Makefile testnet0/1 targets now depend on the "p2pd" binary
- prefixed the testnet command line in the build script message with
  "../../env.sh" so it can find the superproject's "p2pd"
2019-07-11 04:36:07 +02:00
Dustin Brody 7bc2b81e18
~2x state_sim speedup via additional caching in get_crosslink_committee(...) (#316) 2019-07-10 14:00:28 +00:00
Dustin Brody dfa062db7d 0.8.0, continued (#315)
* rename get_genesis_beacon_state(...) to initialize_beacon_state_from_eth1(...); remove unused vestiges get_temporary_block_header(...) and get_empty_block(...); partly align initialize_beacon_state_from_eth1(...) with 0.8.0 version; implement CompactCommittee object type; update process_final_updates(...) to 0.8.0

* mark get_shard_delta(...) as 0.8.0

* mark get_total_active_balance(...) and epoch transition helper functions as 0.8.0

* move FORK_CHOICE_BALANCE_INCREMENT, not in spec anymore, to sole user in fork_choice
2019-07-10 14:23:02 +02:00
Dustin Brody 0a2d09e7fe
Speed up process_crosslinks(...) and get_crosslink_deltas(...) by 10x - 15x in state_sim (#314)
* remove some low-hanging perf silliness from get_winning_crosslink_and_attesting_indices(...) and switch inner/outer loop nesting order to get 10-15x function speedup for 128 and 512 validator cases by avoiding accidentally quadratic behavior, while keeping function signature unchanged and allowing easy ongoing verification of correctness of optimization

* avoid memory allocation with an openarray rather than seq

* update get_crosslink_deltas(...) to 0.8.0

* move non-spec ValidatorSetDeltaFlags from datatypes.nim to sync_protocol; fix mainnet MIN_ATTESTATION_INCLUSION_DELAY preset; update get_attestation_deltas(...) to 0.8.0; for 512 validators, get 90x speedup for get_attestation_deltas(...) from 179s to 2s

* explain purpose of when false portions, so they can be removed once accomplished
2019-07-08 13:19:52 +00:00