* in makeBeaconBlock - use rollback instead
* in tests - this helps state_sim give more accurate data and makes it
30% faster
* fix some usages of raw BeaconState
* update more spec refs in beacon_chain/spec/presets; skip skipped constant sanity checks also from markdown reports' perspectives
* mark skipped as skipped in markdown
* Fix nbench compilation with HashedBeaconState
* Add nbench to tooling
* use newClone - fix 265e01e404 (r425198575)
* Detail advance_slot and hashTreeRoot
* Report throughput
* Fallback for ARM
* windows does not support inline ASM
The work on this was started last week while I was waiting
for a decision on the "Async Snappy" PR. It was prompted by
a failing test in the test suite, where the HashingStream
was inserting some incorrectly padded chunks that affected
the result of `hash_tree_root`. Instead of working around
the problem in the HashingStream, I've decided to implement
a planned optimisation that allows us to remove the hashing
stream altogether.
With the optimisation in place, `hash_tree_root` will now
use only stack memory and only the precise amount neccesary
to build the chunks-merging tree.
* switch state cache to use ref statedata objects to limit memory usage
* more directly initialize ref StateData
* use HashedBeaconState instead of StateData to try to fix memory leak
* switch cache to seq[ref HashedBeaconState]
* remove unused import
Co-authored-by: Ștefan Talpalaru <stefantalpalaru@yahoo.com>
* sync fixes
* fix Status message finalized info
* work around sync starting before initial status exchange
* don't fail block on deposit signature check failure (fixes#989)
* print ForkDigest and Version nicely
* dump incoming blocks
* fix crash when libp2p peer connection is closed
* update chunk size to 16 to work around missing blocks when syncing
* bump libp2p
* bump libp2p
* better deposit skip message
* fix some warnings related to beacon_node splitting; reimplement finalization verification more robustly; improve attestation pool block selection logic
* re-add missing import
* whitelist allowed state transition flags and make rollback/restore naming more consistent
* restore usage of update flags passed into skipAndUpdateState(...) in addition to the potential verifyFinalization flag
* switch rest of rollback -> restore
When replaying state transitions, for the slots that have a block, the
state root is taken from the block. For slots that lack a block, it's
currently calculated using hash_tree_root which is expensive.
Caching the empty slot state roots helps us avoid recalculating this
hash, meaning that for replay, hashes are never calculated. This turns
blocks into fairly lightweight "state-diffs"!
* avoid re-saving state when replaying blocks
* advance empty slots slot-by-slot and save root
* fix sim randomness
* fix sim genesis filename
* introduce `isEpoch` to check if a slot is an epoch slot
* remove incorrect/obsolete comment; deprecate BeaconState state transition functions
* remove deprecated state_transition(state: var BeaconState)
* add specific workarounds for state_transition() and process_slots() to nfuzz_block() and addTestBlock()
* remove near-duplicate code paths: process_slot(), process_slots(), and state_transition() for BeaconState are now wrappers around the HashedBeaconState versions
* convert tests/test_state_transition.nim to use HashedBeaconState
* convert mocking infrastructure and spec_block/epoch_processing tests to use HashedBeaconBlock, and remove thus unused process_slot*(state: var BeaconState)
* ssz: move ref support outside
Instead of allocating ref's inside SSZ, move it to separate helper:
* makes `ref` allocations explicit
* less magic inside SSZ
* `ref` in nim generally means reference whereas SSZ was loading as
value - if a type indeed used references it would get copies instead of
references to a single value on roundtrip which is unexpected
TODO: EF tests would benefit from some refactoring since they all do the
same thing practically..
Co-authored-by: Zahary Karadjov <zahary@gmail.com>
* refactor blook pool caches to directly use TableRef to avoid SSZ decoding, which was consuming 20% of profile on mainnet eth2_network_simulation
* use table's hasKeyOrPut
* bump eth2 spec reference to v0.11.1
* cache whole StateData objects and switch from expensive clear() to cheaper new object instantiation for caching
* remove scaffolding and stop re-assigning to part of StateData object
* 80-character lines
Please see the newly added 'schlesi-dev' Makefile target.
It demonstrates how the log level can be specified for individual topics.
Additionally, when connecting to testnets like 'schlesi' there will be
two additional log files produced in the working directory:
* json-log.txt
* text-log.txt (in the textblocks format)
In BlockPool, we keep the head state around, so it's trivial to restore
the temporary state there and keep going as if nothing happened.
This solves 3 problems:
* stack space - the state copy on mainnet is huge
* GC scanning - using stack space for state slows down the GC
significantly
* reckless copying - the copy itself takes a long time
In state_sim, we'll do the same and allocate on heap - this helps a
little with GC - without it, the collection of the temporary strings
created with `toHex` while printing the json dominates the trace.
* add another check for inconsistent aggregation and committee length, since ncli_transition bypasses process_attestation(...)/check_attestation(...) and calls almost directly into process_epoch(...)
* bump validator functions to v0.11.1 spec references
* bump some spec references to v0.11.1
* poke
* Add "drop by score" ability to PeerPool.
Add tests.
Fix syncmanager queue to start from most fresh data.
* Fix endless cycle at the end of syncing process.
* reduce stack space usage in process_final_updates(...) to avoid fuzzed segfault in https://github.com/status-im/nim-beacon-chain/issues/921
* document motivation behind manually constructing hash_tree_root of a HistoricalBatch
* fix remaining block pool extended validation issues and re-enable first-block-received and block-signature EV checks; enable Merkle validation in beacon_node in eth2_network_simulation; refactor some Merkle proof generation code outside tests/ as a result
* re-enable Merkle validation skipping, since while it works on make eth2_network_simulation, it has issues with local testnet
* tighten already-seen-block blockpool check; move comment closer to conceptually proximate code; queue up maybe-future-valid-blocks as pending to keep libp2p-synchronous interrupt handling time lower
* revert the cleanups, now in a separate PR
* remove the remaining merkle_minimal cleanup remnants, also moved to other PR
* restore PR to only modifying one file after rebasing
* use signatures as summary to compare block contents
* switch signature comparison to be raw byte-wise to ensure no attempts to deserialize it to valid (or not) BLS signatures first
Dustin Brody
Jacek Sieka
Ștefan Talpalaru
Mamy Ratsimbazafy
Zahary Karadjov
Eugene Kabanov
Dmitriy Ryajov
Viktor Kirilov
Diederik Loerakker