nimbus-eth2/tests
Jacek Sieka 5713a3ce4c
performance fixes (#2259)
* performance fixes

* don't mark tree cache as dirty on read-only List accesses
* store only blob in memory for keys and signatures, parse blob lazily
* compare public keys by blob instead of parsing / converting to raw
* compare Eth2Digest using non-constant-time comparison
* avoid some unnecessary validator copying

This branch will in particular speed up deposit processing which has
been slowing down block replay.

Pre (mainnet, 1600 blocks):

```
All time are ms
     Average,       StdDev,          Min,          Max,      Samples,         Test
Validation is turned off meaning that no BLS operations are performed
    3450.269,        0.000,     3450.269,     3450.269,            1, Initialize DB
       0.417,        0.822,        0.036,       21.098,         1400, Load block from database
      16.521,        0.000,       16.521,       16.521,            1, Load state from database
      27.906,       50.846,        8.104,     1507.633,         1350, Apply block
      52.617,       37.029,       20.640,      135.938,           50, Apply epoch block
```

Post:

```
    3502.715,        0.000,     3502.715,     3502.715,            1, Initialize DB
       0.080,        0.560,        0.035,       21.015,         1400, Load block from database
      17.595,        0.000,       17.595,       17.595,            1, Load state from database
      15.706,       11.028,        8.300,      107.537,         1350, Apply block
      33.217,       12.622,       17.331,       60.580,           50, Apply epoch block
```

* more perf fixes

* load EpochRef cache into StateCache more aggressively
* point out security concern with public key cache
* reuse proposer index from state when processing block
* avoid genericAssign in a few more places
* don't parse key when signature is unparseable
* fix `==` overload for Eth2Digest
* preallocate validator list when getting active validators
* speed up proposer index calculation a little bit
* reuse cache when replaying blocks in ncli_db
* avoid a few more copying loops

```
     Average,       StdDev,          Min,          Max,      Samples,         Test
Validation is turned off meaning that no BLS operations are performed
    3279.158,        0.000,     3279.158,     3279.158,            1, Initialize DB
       0.072,        0.357,        0.035,       13.400,         1400, Load block from database
      17.295,        0.000,       17.295,       17.295,            1, Load state from database
       5.918,        9.896,        0.198,       98.028,         1350, Apply block
      15.888,       10.951,        7.902,       39.535,           50, Apply epoch block
       0.000,        0.000,        0.000,        0.000,            0, Database block store
```

* clear full balance cache before processing rewards and penalties

```
All time are ms
     Average,       StdDev,          Min,          Max,      Samples,         Test
Validation is turned off meaning that no BLS operations are performed
    3947.901,        0.000,     3947.901,     3947.901,            1, Initialize DB
       0.124,        0.506,        0.026,      202.370,       363345, Load block from database
      97.614,        0.000,       97.614,       97.614,            1, Load state from database
       0.186,        0.188,        0.012,       99.561,       357262, Advance slot, non-epoch
      14.161,        5.966,        1.099,      395.511,        11524, Advance slot, epoch
       1.372,        4.170,        0.017,      276.401,       363345, Apply block, no slot processing
       0.000,        0.000,        0.000,        0.000,            0, Database block store
```
2021-01-25 13:04:18 +01:00
..
fork_choice Fork choice - almost free pruning - fix #1534 (#1535) 2020-08-26 17:23:34 +02:00
fuzzing Rename binaries; Mimic the original repo layout in the distribution 2020-11-09 11:38:52 +02:00
helpers remove {.inline.} pragmas (#2033) 2020-11-20 11:00:22 +01:00
mocking performance fixes (#2259) 2021-01-25 13:04:18 +01:00
official remove some skipBlsValidation usage/handling in tests (#2258) 2021-01-22 14:29:04 +01:00
simulation pre-emptive duplicate validator detection heuristic 2021-01-21 16:03:02 +02:00
slashing_protection performance fixes (#2259) 2021-01-25 13:04:18 +01:00
spec_block_processing bump 1.0.0rc-0 spec refs to 1.0.0 (#1974) 2020-11-09 14:18:55 +00:00
spec_epoch_processing openarray -> openArray 2020-11-03 23:23:10 +02:00
all_tests.nim Fix #2252 (consistent typing for the 'delay' log property) 2021-01-22 15:09:21 +02:00
all_tests.nim.cfg Rename binaries; Mimic the original repo layout in the distribution 2020-11-09 11:38:52 +02:00
test_attestation_pool.nim remove some skipBlsValidation usage/handling in tests (#2258) 2021-01-22 14:29:04 +01:00
test_beacon_chain_db.nim initial infrastructure for state diffs (#2087) 2021-01-18 22:34:41 +02:00
test_beacon_node.nim Rename binaries; Mimic the original repo layout in the distribution 2020-11-09 11:38:52 +02:00
test_beaconstate.nim Prefer converting int` to uint64 and switch foo.len.uint64 to .len64 (#1375) 2020-07-26 20:55:48 +02:00
test_bitseqs.nim reorder ssz (#1099) 2020-06-03 15:52:02 +02:00
test_block_pool.nim use new stew helpers for assignment (#2172) 2020-12-16 09:37:22 +01:00
test_datatypes.nim Address review comments; Add a test 2020-07-12 21:01:31 +03:00
test_eth1_monitor.nim Rewrite all HTTP(S) web3 URLs to WebSocket URls 2020-11-12 22:29:43 +02:00
test_exit_pool.nim in exit pool, bundle receive messages into beaconblocks (#1812) 2020-10-07 16:57:21 +00:00
test_helpers.nim Generate markdown test reports 2020-03-13 14:38:59 +00:00
test_honest_validator.nim detect already-aggregate-voted condition before attestation pool; add is_aggregator tests (#2170) 2020-12-14 20:58:32 +00:00
test_interop.nim remove v0.12.3 and medalla support (#2038) 2020-11-20 14:49:49 +01:00
test_keystore.nim Fix #1855; Add support for wallet recovery passwords 2020-10-30 23:56:04 +02:00
test_peer_connection.nim more fork-choice fixes (#1388) 2020-07-30 17:48:25 +02:00
test_peer_pool.nim Peer management (#1707) 2020-09-21 18:02:27 +02:00
test_ssz.nim Prefer converting int` to uint64 and switch foo.len.uint64 to .len64 (#1375) 2020-07-26 20:55:48 +02:00
test_ssz_merkleization.nim remove {.inline.} pragmas (#2033) 2020-11-20 11:00:22 +01:00
test_state_transition.nim address some TODOs (#2005) 2020-11-12 19:24:07 +00:00
test_statediff.nim initial infrastructure for state diffs (#2087) 2021-01-18 22:34:41 +02:00
test_sync_manager.nim avoid double deserialization 2020-08-21 11:46:25 +03:00
test_sync_protocol.nim Generate markdown test reports 2020-03-13 14:38:59 +00:00
test_timeutils.nim Fix #2252 (consistent typing for the 'delay' log property) 2021-01-22 15:09:21 +02:00
test_zero_signature.nim performance fixes (#2259) 2021-01-25 13:04:18 +01:00
testblockutil.nim remove some skipBlsValidation usage/handling in tests (#2258) 2021-01-22 14:29:04 +01:00
testutil.nim Use O(n) algorithm in initialize_beacon_state_from_eth1; Avoid unnecessary merkle proofs generation 2020-11-15 21:40:40 +02:00