Replace shuffling function with zrnt version - `get_shuffled_seq` in
particular puts more strain on the GC by allocating superfluous seq's
which turns out to have a significant impact on block processing (when
replaying blocks for example) - 4x improvement on non-epoch, 1.5x on
epoch blocks (replay is done without signature checking)
Medalla, first 10k slots - pre:
```
Loaded 68973 blocks, head slot 117077
All time are ms
Average, StdDev, Min, Max, Samples,
Test
Validation is turned off meaning that no BLS operations are performed
76855.848, 0.000, 76855.848, 76855.848, 1,
Initialize DB
1.073, 0.914, 0.071, 12.454, 7831,
Load block from database
31.382, 0.000, 31.382, 31.382, 1,
Load state from database
85.644, 30.350, 3.056, 466.136, 7519,
Apply block
506.569, 91.129, 130.654, 874.786, 312,
Apply epoch block
```
post:
```
Loaded 68973 blocks, head slot 117077
All time are ms
Average, StdDev, Min, Max, Samples,
Test
Validation is turned off meaning that no BLS operations are performed
72457.303, 0.000, 72457.303, 72457.303, 1,
Initialize DB
1.015, 0.858, 0.070, 11.231, 7831,
Load block from database
28.983, 0.000, 28.983, 28.983, 1,
Load state from database
21.725, 17.461, 2.659, 393.217, 7519,
Apply block
324.012, 33.954, 45.452, 440.532, 312,
Apply epoch block
```
* avoid converting from uint64 to int, and where most feasible, int type conversion at all
* .len.uint64 -> .len64
* fix 32-bit compilation
* try keeping state_sim loop variable/bounds as int for 32-bit Azure
* len64 -> lenu64
* restore EpochRef and flush statecaches on epoch transitions
* more targeted cache invalidation
* remove get_empty_per_epoch_cache(); implement simpler but still faster get_beacon_proposer_index()/compute_proposer_index() approach; add some abstraction layer for accessing the shuffled validator indices cache
* reduce integer type conversions
* remove most of rest of integer type conversion in compute_proposer_index()
* adopt Result[void, string] in place of some bool return signatures
* string -> cstring to reduce memory allocations; ensure all err() strings are constants, with contextual information from higher-level callers
* logScope usage fixes
* homogenize err() reporting convention
* invalid signature in deposit isn't an error
* cleanups
* fix ncli state root check flag
* add block dump to ncli_db
* limit ncli_db benchmark length
* tone down finalization logs
* introduce trusted blocks
We only store blocks whose signature we've verified in the database - as
such, there's no need to check it again, and most importantly, no need
to deserialize the signature when loading from database.
50x startup time improvement, 200x block load time improvement.
* fix rewinding when deposits have invalid signature
* speed up ancestor iteration by avoiding copy
* avoid deserializing signatures for trusted data
* load blocks lazily when rewinding (less memory used)
* chronicles workarounds
* document trustedbeaconblock
* reorder ssz
* split into hash_trees and ssz_serialization, roughly, for hashing and
IO
* move bitseqs into ssz (from stew)
* clean up imports
* docs, imports
* remove incorrect/obsolete comment; deprecate BeaconState state transition functions
* remove deprecated state_transition(state: var BeaconState)
* add specific workarounds for state_transition() and process_slots() to nfuzz_block() and addTestBlock()
* ssz: move ref support outside
Instead of allocating ref's inside SSZ, move it to separate helper:
* makes `ref` allocations explicit
* less magic inside SSZ
* `ref` in nim generally means reference whereas SSZ was loading as
value - if a type indeed used references it would get copies instead of
references to a single value on roundtrip which is unexpected
TODO: EF tests would benefit from some refactoring since they all do the
same thing practically..
Co-authored-by: Zahary Karadjov <zahary@gmail.com>
In BlockPool, we keep the head state around, so it's trivial to restore
the temporary state there and keep going as if nothing happened.
This solves 3 problems:
* stack space - the state copy on mainnet is huge
* GC scanning - using stack space for state slows down the GC
significantly
* reckless copying - the copy itself takes a long time
In state_sim, we'll do the same and allocate on heap - this helps a
little with GC - without it, the collection of the temporary strings
created with `toHex` while printing the json dominates the trace.
* Initial implementation of runtime bls skipping.
Add libnfuzz skipBLSValidation handling, check that it propagates.
* Rename skipBLSValidation -> skipBlsValidation, start skipStateRootValidation
* Replace skipValidation flags with more granular flags.
Also added skipBlockParentRootValidation flag
Mainly replaced with skipBlsValidation but also StateRoot or
BlockParentRootValidation flags where appropriate.
* Adjust interop test to pass when skipping merkle validation.
* Stop skipping validation for mainchain_monitor.
* Remove comment.
* Also skipMerkleValidation for test_beacon_chain_db.
tersec
Jacek Sieka
Zahary Karadjov
Nathaniel Jensen
kdeme