Commit Graph

912 Commits

Author SHA1 Message Date
Mamy Ratsimbazafy 621c2e38a6
Separate validator duties from the Beacon Node (#976)
* Separate validator duties from the Beacon Node

* remove duplicate MaxEmptySlotCOunt

* imports

* have beacnde properly compile
2020-05-06 13:23:45 +00:00
Jacek Sieka a56e18bb0c
collect garbage at end of each slot (#975) 2020-05-06 12:01:19 +02:00
tersec c548b90a0e
default preset update; rm process_slots(BeaconState); some BLS skipping disabled (#963)
* re-enable BLS validation for block sanity tests

* update default preset to mainnet; remove process_slots(state: var BeaconState)

* bump spec ref to v0.11.1

* bump all v0.11.1 spec references except the actually changed is_valid_indexed_attestation(...) in spec/beaconstate to v0.11.2

* remove stack-risky/low-ROI doAssert
2020-05-04 21:07:18 +00:00
Jacek Sieka 01e9df97cb
cleanups (#962)
* remove broken serialized_sizes
* actually use sszdump module
* avoid bitops
* fix stack_sizes module name
2020-05-04 07:38:14 +02:00
Jacek Sieka 2449d4b479
cache empty slot state root (#961)
When replaying state transitions, for the slots that have a block, the
state root is taken from the block. For slots that lack a block, it's
currently calculated using hash_tree_root which is expensive.

Caching the empty slot state roots helps us avoid recalculating this
hash, meaning that for replay, hashes are never calculated. This turns
blocks into fairly lightweight "state-diffs"!

* avoid re-saving state when replaying blocks
* advance empty slots slot-by-slot and save root
* fix sim randomness
* fix sim genesis filename
* introduce `isEpoch` to check if a slot is an epoch slot
2020-05-03 19:44:04 +02:00
Jacek Sieka a3e098cf92
block pool simulator (#956)
* block pool simulator

like state_sim, but more
2020-05-01 17:51:24 +02:00
tersec cf8e90615a
More state_transition unification (#953)
* remove incorrect/obsolete comment; deprecate BeaconState state transition functions

* remove deprecated state_transition(state: var BeaconState)

* add specific workarounds for state_transition() and process_slots() to nfuzz_block() and addTestBlock()
2020-04-30 18:27:17 +02:00
Jacek Sieka e65f5c86da
vendor: bump (#954) 2020-04-30 18:27:04 +02:00
tersec cd939d4606
fix spec/beaconstate compile error in the presence of other PRs' SSZ tightening (#952) 2020-04-30 09:01:55 +00:00
tersec 3a56ddc5c4
begin removing plain BeaconState versions of state transition functions (#951)
* remove near-duplicate code paths: process_slot(), process_slots(), and state_transition() for BeaconState are now wrappers around the HashedBeaconState versions

* convert tests/test_state_transition.nim to use HashedBeaconState

* convert mocking infrastructure and spec_block/epoch_processing tests to use HashedBeaconBlock, and remove thus unused process_slot*(state: var BeaconState)
2020-04-30 08:44:19 +02:00
Jacek Sieka c74ba5c0c6
ssz: move ref support outside (#943)
* ssz: move ref support outside

Instead of allocating ref's inside SSZ, move it to separate helper:

* makes `ref` allocations explicit
* less magic inside SSZ
* `ref` in nim generally means reference whereas SSZ was loading as
value - if a type indeed used references it would get copies instead of
references to a single value on roundtrip which is unexpected

TODO: EF tests would benefit from some refactoring since they all do the
same thing practically..

Co-authored-by: Zahary Karadjov <zahary@gmail.com>
2020-04-29 22:12:07 +02:00
cheatfate 49cc9a9961 Fix updateStatus should not raise exceptions. 2020-04-29 19:21:18 +03:00
tersec 57aba5d3a6
Switch block pool caches from BeaconChainDB to TableRefs (#945)
* refactor blook pool caches to directly use TableRef to avoid SSZ decoding, which was consuming 20% of profile on mainnet eth2_network_simulation

* use table's hasKeyOrPut

* bump eth2 spec reference to v0.11.1

* cache whole StateData objects and switch from expensive clear() to cheaper new object instantiation for caching

* remove scaffolding and stop re-assigning to part of StateData object

* 80-character lines
2020-04-29 16:58:44 +02:00
Zahary Karadjov 667bdf066a Bump Chronicles to fix a minor bug; Add a placeholder for disabled topics 2020-04-28 13:49:30 +03:00
Zahary Karadjov 7237a370ba Implement https://github.com/status-im/nim-beacon-chain/issues/872
Please see the newly added 'schlesi-dev' Makefile target.
It demonstrates how the log level can be specified for individual topics.

Additionally, when connecting to testnets like 'schlesi' there will be
two additional log files produced in the working directory:

* json-log.txt
* text-log.txt (in the textblocks format)
2020-04-28 13:49:30 +03:00
Jacek Sieka 28d6cd2524
avoid memory allocations and copies when loading states (#942)
* rolls back some of the ref changes
* adds utility to calculate stack sizes
* works around bugs in nim exception handling and rvo
2020-04-28 10:08:32 +02:00
Zahary Karadjov 80b538452e Updated the custom preset loader to the latest spec; Working makefile target for connecting to the Schlesi testnet 2020-04-27 23:22:54 +03:00
Viktor Kirilov dcc70c5259 resolved final comment from review 2020-04-27 22:10:35 +03:00
Viktor Kirilov 8ef28c905d fixed commets from review 2020-04-27 22:10:35 +03:00
Viktor Kirilov 5ecba6df49 broadcasting and listening to the '_snappy' topics as well 2020-04-27 22:10:35 +03:00
Jacek Sieka 898df9ba45
kvstore: port to nim-eth (#938) 2020-04-27 18:36:28 +02:00
tersec 7790644e52
remove a pointless hash_tree_root(BeaconState) per node per proposed block (#933)
* remove a pointless hash_tree_root(BeaconBlock)

* use ref with putState
2020-04-27 12:47:49 +02:00
Jacek Sieka 03a147ab8d
avoid state copy in state transition (#930)
In BlockPool, we keep the head state around, so it's trivial to restore
the temporary state there and keep going as if nothing happened.

This solves 3 problems:
* stack space - the state copy on mainnet is huge
* GC scanning - using stack space for state slows down the GC
significantly
* reckless copying - the copy itself takes a long time

In state_sim, we'll do the same and allocate on heap - this helps a
little with GC - without it, the collection of the temporary strings
created with `toHex` while printing the json dominates the trace.
2020-04-26 21:13:33 +02:00
tersec be475a82d7
catch invalid 0-raw-byte Bitlist SSZ inputs during parsing (#936)
* catch invalid 0-raw-byte Bitlist SSZ inputs during parsing

* bump a couple more spec refs to v0.11.1
2020-04-26 16:26:53 +00:00
Zahary Karadjov 4e9fa51ae9 Introduce BeaconNodeRef and use it in all the right places 2020-04-26 13:04:53 +03:00
Zahary Karadjov fdcbfdff05 Pass the test suite with a BeaconState ref type 2020-04-26 13:04:53 +03:00
Zahary Karadjov 740b76d152 Spec-compliant implementation of Eth1 monitoring; Eth1-enabled local sim
BEWARE! This commit will trigger a stack overflow during local sim
2020-04-26 13:04:53 +03:00
tersec 65353bab76
add another check for inconsistent aggregation and committee length (#927)
* add another check for inconsistent aggregation and committee length, since ncli_transition bypasses process_attestation(...)/check_attestation(...) and calls almost directly into process_epoch(...)

* bump validator functions to v0.11.1 spec references

* bump some spec references to v0.11.1

* poke
2020-04-24 09:16:40 +02:00
Jacek Sieka 494ffb63ce
eh fixes (#926)
* work around improbable exceptions in metrics / chronos
* fix unnecessary lookup in block pool
2020-04-24 09:16:11 +02:00
tersec 7bc18423fa
detect improper 0-length lists of variable-sized objects in SSZ reading (#928)
* detect improper 0-length lists of variable-sized objects in SSZ reading
2020-04-23 19:39:23 +00:00
Viktor Kirilov 0079242457 again exported the right overloads for serializing hashes properly to json (and not as byte arrays) and updating nim-eth 2020-04-23 20:46:13 +03:00
Viktor Kirilov 4c66999a43 removing the json serialization change - fails to compile with -d:testnet_servers_image and LOG_LEVEL=TRACE and isn't worth the time at the moment 2020-04-23 20:46:13 +03:00
Viktor Kirilov 1575ded589 exported the right overloads for serializing hashes properly to json (and not as byte arrays) 2020-04-23 20:46:13 +03:00
Viktor Kirilov f3d0ae35f6 added the attestation bitfield to the bootstrap enr 2020-04-23 20:46:13 +03:00
Eugene Kabanov be89a3c54d
Add "drop by score" ability to PeerPool. (#917)
* Add "drop by score" ability to PeerPool.
Add tests.
Fix syncmanager queue to start from most fresh data.

* Fix endless cycle at the end of syncing process.
2020-04-23 17:31:00 +02:00
tersec 7cd4b0bfae
reduce stack space usage in process_final_updates(...) (#925)
* reduce stack space usage in process_final_updates(...) to avoid fuzzed segfault in https://github.com/status-im/nim-beacon-chain/issues/921

* document motivation behind manually constructing hash_tree_root of a HistoricalBatch
2020-04-23 13:39:38 +02:00
tersec 7e94482409
avoid rewindStateData/state_transition in fast-path (#919)
* avoid rewindStateData/state_transition totally in fast-pathed updateStateData calls

* document motivation behind BlockPool.getStateDataCached(...) and clarify naming
2020-04-23 11:59:29 +02:00
Jacek Sieka 6729d3c032
kvstore: fix raising, be harsher on database errors (#923)
* kvstore: fix raising, be harsher on database errors

* bump stew/serialization
2020-04-23 08:27:35 +02:00
Jacek Sieka 65ca74c980 req: cap requested blocks better
also cap blocks in roots request
2020-04-22 12:09:26 +03:00
Jacek Sieka ed74770451
spec: regulate exceptions (#913)
* spec: regulate exceptions

* a few more simple raises
2020-04-22 07:53:02 +02:00
tersec bdea75e4c3
Restore all blockpool extended validation checks (#886)
* fix remaining block pool extended validation issues and re-enable first-block-received and block-signature EV checks; enable Merkle validation in beacon_node in eth2_network_simulation; refactor some Merkle proof generation code outside tests/ as a result

* re-enable Merkle validation skipping, since while it works on make eth2_network_simulation, it has issues with local testnet

* tighten already-seen-block blockpool check; move comment closer to conceptually proximate code; queue up maybe-future-valid-blocks as pending to keep libp2p-synchronous interrupt handling time lower

* revert the cleanups, now in a separate PR

* remove the remaining merkle_minimal cleanup remnants, also moved to other PR

* restore PR to only modifying one file after rebasing

* use signatures as summary to compare block contents

* switch signature comparison to be raw byte-wise to ensure no attempts to deserialize it to valid (or not) BLS signatures first
2020-04-21 18:52:53 +02:00
Jacek Sieka da988e4e40 more block range request updates
* handle skip == 0 gracefully
* avoid memory allocation at expense of more complex API
* add more tests
* log block request results
2020-04-21 11:07:27 +03:00
Jacek Sieka fccce85f0d simplify block request
this matches the intended spec behaviour
2020-04-21 11:07:27 +03:00
Jacek Sieka 2c2de2b35e
pull out validator duties to simplift onSlot handler (#910) 2020-04-20 19:28:12 +02:00
tersec 8a72ae89b9
fix mainnet finalization (#906)
* fix mainnet finalization and swith eth2_network_simulation to a kind of small-mainnet profile

* Fix slot reference in trace logging

* bump a couple of spec refs from v0.11.0 to v0.11.1

* bump another spec ref to v0.11.1, one more try at Jenkins test vector download CI issue

* fix other slot reference in trace logging and skip past single-block/multi-slot gaps to re-approach from ancestry side by state_transitioning, by requiring exact match on both root hash and slot for fast path

* make more precise the fast path condition

* redo logic to make uniform with BeaconChainDB; fix chronos deprecation warning

* revert not-working replacement of deprecated chronos futures `or`

* switch testnet1 to mainnet
2020-04-20 19:27:52 +02:00
Eugene Kabanov 3d42da90a8
Syncing. (#909) 2020-04-20 16:59:18 +02:00
Jacek Sieka edecce1751
disable aggregation when no validators are connected (#908)
* if we don't have validators, don't consider aggregation work
* if we do have validators, don't aggregate when we're out of sync
* when we do aggregate, use a fresh state, and not one from before
sleeping
2020-04-20 14:20:53 +02:00
tersec 480c11f8aa
fix remaining block sanity test vectors (#902)
* fix remaining block sanity test vectors

* bump epoch processing spec refs to v0.11.1
2020-04-18 00:09:34 +02:00
Dustin Brody 164d171bd9
revert previous commit 2020-04-17 20:27:42 +02:00
Dustin Brody ef92529feb
fix remaining block sanity test vectors 2020-04-17 20:25:18 +02:00