Commit Graph

59 Commits

Author SHA1 Message Date
Zahary Karadjov cdff79ec6d More sync fixes
* Fix IncompleteData issues brought by the new spec-compliant stream closing
* Fix logic errors in the sync algorithms
2019-09-10 09:49:54 -04:00
Zahary Karadjov 886b92319f Implement the response size limits 2019-09-10 09:49:54 -04:00
Zahary Karadjov a83aa83644 Working BeaconSync
Changes:

* Do not send separate network packets for response codes and msg
  len prefixes

* Close streams according to the spec

* Implement more timeouts according to the spec

* Make hello requests during syncing to update our knowledge of
  the head block of the other peer.
2019-09-10 09:49:54 -04:00
Zahary Karadjov 4a54fb4103 Cleaned up obsolete BeaconSync code; Added some open questions regarding fetchAncestorBlocks 2019-09-10 09:49:54 -04:00
Zahary Karadjov 9dec05f9c9 Sending chunked responses 2019-09-10 09:49:54 -04:00
Zahary Karadjov 2bbfa8c877 Changes related to the new ETH interop spec
* Hello is no longer a handshake message
  (all handshakes related code was deleted for clarity)

* Deal with the single-parameter inlining defined in the new spec
2019-09-10 09:49:54 -04:00
Jacek Sieka 5f58af2112
fix ssz list, protocol names (#405) 2019-09-08 16:54:31 +02:00
Eugene Kabanov bb0e9de09d [WIP] Sync which using latest network specification. (#390)
* Fixed getBeaconBlocks() and getRecentBeaconBlocks() to use BlockPool, not db.

* Got the sync_protocol to compiling state; Removed all obsolete RPC calls
2019-09-08 15:08:44 +02:00
zah 93cdb43f1e Integrate nim-metrics and add some metrics from the ETH2 spec (#394) 2019-09-07 19:48:05 +02:00
Zahary Karadjov f5db2dd7b3
Fix the build with the latest Chronos 2019-08-15 20:43:06 +02:00
Zahary Karadjov 2fca9465e7
Fix the build with RLPx 2019-08-15 20:43:05 +02:00
Zahary Karadjov 5ce50b3aca
Implement the latest networking spec
https://github.com/ethereum/eth2.0-specs/pull/1328
2019-08-07 05:22:28 +03:00
Zahary Karadjov 398ea55801
Implement the latest SSZ specification and integrate the official SSZ test suite 2019-08-05 03:18:48 +03:00
Dustin Brody 0a2d09e7fe
Speed up process_crosslinks(...) and get_crosslink_deltas(...) by 10x - 15x in state_sim (#314)
* remove some low-hanging perf silliness from get_winning_crosslink_and_attesting_indices(...) and switch inner/outer loop nesting order to get 10-15x function speedup for 128 and 512 validator cases by avoiding accidentally quadratic behavior, while keeping function signature unchanged and allowing easy ongoing verification of correctness of optimization

* avoid memory allocation with an openarray rather than seq

* update get_crosslink_deltas(...) to 0.8.0

* move non-spec ValidatorSetDeltaFlags from datatypes.nim to sync_protocol; fix mainnet MIN_ATTESTATION_INCLUSION_DELAY preset; update get_attestation_deltas(...) to 0.8.0; for 512 validators, get 90x speedup for get_attestation_deltas(...) from 179s to 2s

* explain purpose of when false portions, so they can be removed once accomplished
2019-07-08 13:19:52 +00:00
Jacek Sieka b9e62a1f6c
std_shims -> stew 2019-07-07 11:53:58 +02:00
Dustin Brody e4321dc4ed Initial frozen phase 0/v0.8.0 spec (#305)
* rename slot_to_epoch(...) to compute_epoch_of_slot(...)

* rename aggregation_bitfield/custody_bitfield fields to aggregation_bits/custody_bits; rename convert_to_indexed(...) to get_indexed_attestation(...); mark BeaconBlockHeader as 0.8.0; update minimal preset MIN_ATTESTATION_INCLUSION_DELAY/time parameters in general to 0.8

* fix beacon node compilation; it used slightly different capitalizations of slot_to_epoch/slotToEpoch

* mark integer_squareroot(...), Deposit, VoluntaryExit, and Transfer as 0.8.0; rename get_epoch_start_slot(...) to compute_start_slot_of_epoch(...)

* add new Domain alias for uint64; rename bls_domain(...) to compute_domain(...), MAX_INDICES_PER_ATTESTATION to MAX_VALIDATORS_PER_COMMITTEE, and SignatureDomain to DomainType; update get_domain(...) to 0.8.0

* update/mark initiate_validator_exit(...) and Crosslink as 0.8.0; rename BeaconState.latest_block_roots -> BeaconState.block_roots; mark HistoricalBatch as 0.8.0
2019-07-01 09:53:42 +02:00
Zahary Karadjov c751285112
Implement all libp2p_native response codes as specified in the latest proposal 2019-06-24 05:34:22 +03:00
Zahary Karadjov e228c2dbcb
Implement the even/odd request ID scheme; Handle more edge cases; Break the cyclic imports 2019-06-24 05:34:22 +03:00
Zahary Karadjov 877b22cfb8
Share more code between the libp2p backends 2019-06-24 05:34:21 +03:00
Zahary Karadjov 87601a5eae
Share more code between the libp2p backends 2019-06-24 05:34:21 +03:00
Zahary Karadjov f4a96bc3f3
[WIP] Restore the Lib2P2 builds and implement the latest wire spec
Depends on https://github.com/status-im/nim-eth/pull/54
2019-06-24 05:34:21 +03:00
Dustin Brody e9180c490e
more 0.7.0 updates (#287)
* update Deposit and BeaconBlockHeader to 0.7.0

* update get_total_balance(...) and get_base_reward(...) to 0.7.0

* rename initial value constant BLS_WITHDRAWAL_PREFIX_BYTE to BLS_WITHDRAWAL_PREFIX

* mark minimal presets as entirely migrated to 0.7.0; update process_rewards_and_penalties(...) to 0.7.0

* specify that targeting 0.7.0; adopt 0.7.0 refactoring of bls_domain/get_domain to update get_domain(...) to 0.7.0; mark that mainnet constant presets and block header processing are 0.7.0-consistent; update process_justification_and_finalization to 0.7.0 to fix potential underflow errors

* address mratsim's comment to copy whole spec comment
2019-06-14 16:21:04 +00:00
Dustin Brody 2a6e64d3f9
0.7.0 updates including semantic changes (#285)
* update get_previous_epoch(...)

* update BeaconBlock to 0.7.0 by renaming previous_block_root field to parent_root
2019-06-14 13:50:47 +00:00
Yuriy Glukhov 10c7920b27 Proto changes to facilitate backward sync (#271)
* Proto changes to facilitate backward sync

* Update to latest spec types in sync proto

* Use blockpool for more straightforward block headers collection

* Added BlockPool.getRef

* Update beacon_chain/sync_protocol.nim

Co-Authored-By: Jacek Sieka <arnetheduck@gmail.com>
2019-06-10 07:13:53 -04:00
Jacek Sieka bd4893d2e8 head block selection fixes (#259)
* track justified and head blocks correctly in presence of blank slots
* fix new block state root
* keep list of viable heads instead of walking children
* disable json bitfield deserializer (needs a corresponding serializer)
* fix handshake best & finalized root
2019-05-01 12:19:29 +03:00
Yuriy Glukhov ddcdb63430
Deduplicate blocks in getAncestorBlocks (#254)
* Deduplicate blocks in getAncestorBlocks

* Limit ancestor blocks in getAncestorBlocks
2019-04-26 09:21:46 +03:00
Yuriy Glukhov b916b1e3e4
Slightly more validations 2019-04-10 15:56:22 +03:00
Yuriy Glukhov 54f894fc7e
Chunked sync second try
This reverts commit cdf9baf5e6.
2019-04-10 15:35:52 +03:00
Zahary Karadjov cdf9baf5e6 Revert "Chunked sync"
This reverts commit c84d930304.
2019-04-09 02:19:33 +03:00
Yuriy Glukhov c84d930304 Chunked sync 2019-04-09 00:18:53 +03:00
Zahary Karadjov 44d9f7d6c9 Use proper RPC calls when obtaining missing blocks 2019-03-28 16:54:50 +02:00
Zahary Karadjov 406df9d44f More chatty syncing 2019-03-27 18:13:39 +02:00
Zahary Karadjov c0fa229254 Improved error handling in the networking code 2019-03-26 17:40:20 +02:00
Yuriy Glukhov e4c10a31c9 Use latest state from the DB in sync. Fixed off by one in block roots (#203) 2019-03-25 14:52:23 -06:00
Zahary Karadjov 70cb84ef93 Fix the build with the latest nim-eth 2019-03-25 19:57:05 +02:00
Jacek Sieka 1b0e67c88c
ssz: update to 0.5.1:ish (#202)
* ssz: update to 0.5.1:ish
* slightly fewer seq allocations
* still a lot of potential for optimization
* fixes #174
* ssz: avoid reallocating leaves (logN merkle impl)
2019-03-25 10:46:31 -06:00
Zahary Karadjov 6bb38095c9 Command-line and data storage handling for testnets 2019-03-18 12:45:29 +02:00
Dustin Brody f36c2d86dc begin 0.5.0 spec update (#179)
* begin 0.5.0 spec update: parent_root -> previous_block_root, BeaconState.justified_epoch -> BeaconState.current_justified_epoch, DOMAIN_PROPOSAL -> DOMAIN_BEACON_BLOCK, temporarily rename BeaconBlockHeader to BeaconBlockHeaderRLP to allow for gradual re-merging without disrupting RLP; mark a few unchanged functions and data types, implement get_temporary_block_header/get_empty_block/should_update_validator_registry/processBlockHeader/cacheState; update a few others

* a dozen or so more trivial mostly comment changes, finding more unchanged parts of 0.5.0

* several more trivial changes; goal is to reduce noise around the upcoming substantial changes (epoch processing, etc)
2019-03-16 13:52:37 -06:00
Ștefan Talpalaru de295619be
assert() -> doAssert() 2019-03-14 00:04:43 +01:00
Zahary Karadjov 562eafe124 Finish the Spec/Epoch types refactoring 2019-03-13 23:23:01 +02:00
Dustin Brody 1479bae22f implement distinct Epoch type and continue 0.4.0 spec update (#173)
* implement distinct Epoch type
* update two more spec 0.4.0 markers, leaving just deposit processing
2019-03-12 19:46:44 -06:00
Dustin Brody 50a0948bc0 Mostly convert to using distinct Slot type (#172) 2019-03-12 16:21:32 -06:00
Mamy André-Ratsimbazafy 6dea18c140
Squashed commit of the following:
commit bed6909f33
Author: Mamy André-Ratsimbazafy <mamy_github@numforge.co>
Date:   Mon Mar 11 12:22:35 2019 +0100

    Cleanup imports

commit 241f62ed8d
Author: Mamy André-Ratsimbazafy <mamy_github@numforge.co>
Date:   Mon Mar 11 12:06:00 2019 +0100

    Group beacon types to prevent recursive imports
2019-03-12 16:03:14 +01:00
Zahary Karadjov 903cb8a8b5 A LibP2P-based networking stack;
To enable it, comment out the 'withLibp2p' line in nim.cfg

The history was squashed in order to remove an accidentally
commited binary file.

Other changes:

* SSZ was adapted to use the common serialization framework

* gossibsup.subscribe is not using async handlers at the moment
  and this allowed me to simplify it
2019-03-12 13:36:55 +02:00
Yuriy Glukhov b587455e7b Use BlockPool for slot->blocks mapping (#166) 2019-03-11 09:38:36 -06:00
Yuriy Glukhov 38f48753b2 Comments on #117 addressed (#154) 2019-03-05 07:55:09 -06:00
Yuriy Glukhov 4c31f50889
Comments addressed 2019-03-01 22:09:20 +02:00
Yuriy Glukhov bed6510da3
Initial full sync impl 2019-03-01 19:39:38 +02:00
Dustin Brody 132f1147a2 spec 0.3.0 updates: EPOCH_LENGTH -> SLOTS_PER_EPOCH, SlotNumber -> Slot, EpochNumber -> Epoch (#127)
* spec 0.3.0 updates: EPOCH_LENGTH -> SLOTS_PER_EPOCH, SlotNumber -> Slot, EpochNumber -> Epoch
2019-02-19 19:33:58 -06:00
Mamy Ratsimbazafy 11da02c34f
Asyncdispatch2 rebranding to chronos (#84)
* Asyncdispatch2 rebranding to chronos

* nim-eth and nim-chronos part of nimble packages

* fix package name
2019-02-06 18:56:04 +01:00