status-im/nimbus-eth2
2
0
Fork 0
mirror of https://github.com/status-im/nimbus-eth2.git synced 2025-01-11 23:04:26 +00:00
Code Issues Projects Releases Wiki Activity
6,992 Commits 108 Branches 116 Tags 245 MiB
Commit Graph

569 Commits

Author SHA1 Message Date
Dustin Brody
490e4c7666 don't used sorted() in get_indexed_attestation() 2021-01-30 00:23:02 +02:00
tersec
1bdbf099cc
use IntSet rather than HashSet[ValidatorIndex] (#2267) 
* use IntSet rather than HashSet[ValidatorIndex]

* add bounds check before uint64 -> int conversion

* use intsets in block transitions

* remove superfluous Nim issue explanation/reference
 2021-01-26 12:52:00 +01:00
Jacek Sieka
5ca10d3de2 avoid copying keys for trusted signatures 
this removes `is_valid_indexed_attestion` from performance benchmarks
when replaying blocks
 2021-01-25 19:48:02 +02:00
Jacek Sieka
5713a3ce4c
performance fixes (#2259) 
* performance fixes

* don't mark tree cache as dirty on read-only List accesses
* store only blob in memory for keys and signatures, parse blob lazily
* compare public keys by blob instead of parsing / converting to raw
* compare Eth2Digest using non-constant-time comparison
* avoid some unnecessary validator copying

This branch will in particular speed up deposit processing which has
been slowing down block replay.

Pre (mainnet, 1600 blocks):

```
All time are ms
     Average,       StdDev,          Min,          Max,      Samples,         Test
Validation is turned off meaning that no BLS operations are performed
    3450.269,        0.000,     3450.269,     3450.269,            1, Initialize DB
       0.417,        0.822,        0.036,       21.098,         1400, Load block from database
      16.521,        0.000,       16.521,       16.521,            1, Load state from database
      27.906,       50.846,        8.104,     1507.633,         1350, Apply block
      52.617,       37.029,       20.640,      135.938,           50, Apply epoch block
```

Post:

```
    3502.715,        0.000,     3502.715,     3502.715,            1, Initialize DB
       0.080,        0.560,        0.035,       21.015,         1400, Load block from database
      17.595,        0.000,       17.595,       17.595,            1, Load state from database
      15.706,       11.028,        8.300,      107.537,         1350, Apply block
      33.217,       12.622,       17.331,       60.580,           50, Apply epoch block
```

* more perf fixes

* load EpochRef cache into StateCache more aggressively
* point out security concern with public key cache
* reuse proposer index from state when processing block
* avoid genericAssign in a few more places
* don't parse key when signature is unparseable
* fix `==` overload for Eth2Digest
* preallocate validator list when getting active validators
* speed up proposer index calculation a little bit
* reuse cache when replaying blocks in ncli_db
* avoid a few more copying loops

```
     Average,       StdDev,          Min,          Max,      Samples,         Test
Validation is turned off meaning that no BLS operations are performed
    3279.158,        0.000,     3279.158,     3279.158,            1, Initialize DB
       0.072,        0.357,        0.035,       13.400,         1400, Load block from database
      17.295,        0.000,       17.295,       17.295,            1, Load state from database
       5.918,        9.896,        0.198,       98.028,         1350, Apply block
      15.888,       10.951,        7.902,       39.535,           50, Apply epoch block
       0.000,        0.000,        0.000,        0.000,            0, Database block store
```

* clear full balance cache before processing rewards and penalties

```
All time are ms
     Average,       StdDev,          Min,          Max,      Samples,         Test
Validation is turned off meaning that no BLS operations are performed
    3947.901,        0.000,     3947.901,     3947.901,            1, Initialize DB
       0.124,        0.506,        0.026,      202.370,       363345, Load block from database
      97.614,        0.000,       97.614,       97.614,            1, Load state from database
       0.186,        0.188,        0.012,       99.561,       357262, Advance slot, non-epoch
      14.161,        5.966,        1.099,      395.511,        11524, Advance slot, epoch
       1.372,        4.170,        0.017,      276.401,       363345, Apply block, no slot processing
       0.000,        0.000,        0.000,        0.000,            0, Database block store
```
 2021-01-25 13:04:18 +01:00
Jacek Sieka
e7f2735271
fix broken metrics during replay (#2090) 
* move metrics out of state transition
* add validator count metric
* remove expensive beacon_current_validators, beacon_previous_validators
metrics (they should be reimplemented with cache), add cheap
beacon_active_validators to approximate
* remove unused validator count metrics
* tidy imports/defects
 2020-11-27 23:16:13 +01:00
Zahary Karadjov
a995cdd3d9
Bugfix: fix an incorrect balance increase during genesis state generation 2020-11-24 23:30:26 +02:00
tersec
9e716b32bd
address some XDeclaredButNotUsed hints (#2028) 2020-11-17 11:14:53 +01:00
Zahary Karadjov
b022dc4d1f Use O(n) algorithm in initialize_beacon_state_from_eth1; Avoid unnecessary merkle proofs generation 2020-11-15 21:40:40 +02:00
tersec
b9e122a60c
address some TODOs (#2005) 2020-11-12 19:24:07 +00:00
tersec
271df8b604
bump 1.0.0rc-0 spec refs to 1.0.0 (#1974) 2020-11-09 14:18:55 +00:00
Jacek Sieka
95f5f76180
Datatype cleanup (#1953) 
* clear up spec todo

* test fix

* remove unnecessary toSszType

* type

* one more
 2020-11-04 21:52:47 +00:00
Zahary Karadjov
14b2d4324d openarray -> openArray 2020-11-03 23:23:10 +02:00
Jacek Sieka
ca63d48b82
tone down validator exit logging (#1939) 
Validators exiting is normal, no need to scream about it

* avoid reallocating seq on big exit queue
* avoid fetching state cache when updating head (it's rarely needed)
* remove incorrectly implemented live validator counts (avoids memory
allocs)
 2020-11-02 18:34:23 +01:00
Jacek Sieka
7c0b4d28d2 speed up reward/penalty calculation 
Calculating rewards/penalties is slow due to how we compute sets of
attestations validators then use the sets for inclusion checks, to see
who attested. The dominant function during validated block processing /
epoch processing is hash set building and lookup.

This PR inverts the flow by removing the sets and creating a single
large validator status list, then applying all relevant state
attestations, then updating rewards and penalties.

This provides a 10x speedup to epoch processing which in turn speeds up
both empty slot and block processing - for example, on startup, we
replay all non-finalized blocks to prime fork choice - the same when
validating attestations or replaying states on reorg.
 2020-10-23 19:23:36 +03:00
Jacek Sieka
499e5ca991
misc memory and perf fixes (#1899) 
* misc memory and perf fixes

* use EpochRef for attestation aggregation
* compress effective balances in memory (medalla unfinalized: 4gb ->
1gb)
* avoid hitting db when rewinding to head or clearance state
* avoid hitting db when blocks can be applied to in-memory state -
speeds up startup considerably
* avoid storing epochref in fork choice
* simplify and speed up beacon block creation flow - avoids state reload
thanks to head rewind optimization
* iterator-based committee and attestation participation help avoid lots
of small memory allocations throughout epoch transition (40% speedup on
epoch processing, for example during startup)

* add constant for threshold
 2020-10-22 12:53:33 +02:00
tersec
e87e0ef179
update to 1.0.0-rc.0 message ID; unexport funcs; enable test in mainnet; remove unused subscribe-with-nondummy-handler function; fix gossip received metric (#1884) 2020-10-20 08:54:11 +00:00
tersec
3ee2dd8da4
p2p-interface spec ref bump (except non-updated places) (#1862) 2020-10-12 14:37:14 +00:00
tersec
1994ffe5a0
update 130+ spec references from v0.12.3 to v1.0.0-rc1 (#1854) 2020-10-12 08:59:24 +00:00
tersec
b79e5f8af5
update nim-beacon-chain to nimbus-eth2 in beacon_chain/, ncli/, tests/, and README.md (#1843) 2020-10-08 19:02:05 +00:00
tersec
c0aa2f1e58
update process_slashings() to use PROPORTIONAL_SLASHING_MULTIPLIER (#1841) 2020-10-08 18:00:07 +00:00
Zahary Karadjov
aed291128a Add support for starting from weak subjectivity checkpoints 
Also removes the `genesis.ssz` file stored in the data folder.
The `medalla-fast-sync` target has been adapted to use the new features.
 2020-10-07 09:32:03 +03:00
Mamy Ratsimbazafy
0280d6c73e
Revisiting log levels (#1788) 
* Update log level - https://github.com/status-im/nim-beacon-chain/issues/1779 https://github.com/status-im/nim-beacon-chain/issues/1785

* Address review comments

* Document the logging strategy [skip ci]
 2020-10-01 20:56:42 +02:00
tersec
f96ad87d28
switch another 50+ spec refs from v0.12.2 to v0.12.3 (#1749) 2020-09-25 11:52:50 +00:00
tersec
6398a43cc1
update 120+ beacon_chain and validator spec refs from v0.12.2 to v0.12.3 (#1740) 2020-09-24 19:04:10 +02:00
tersec
b3b578501a
mark comments documenting functions as a whole as such (#1613) 2020-09-08 08:54:55 +00:00
tersec
e4a43f7628
address issue #1580 (#1600) 
* address issue #1580

* Update beacon_chain/spec/beaconstate.nim

Co-authored-by: Jacek Sieka <jacek@status.im>
 2020-09-02 16:16:25 +00:00
Jacek Sieka
9244ae7a38 more speedups 
* evaluate block attestations under the epochref of the block - this is
what the state transition function does
* avoid copying attestation seq unnecessarily
* avoid unnecessary hashset for unslashed indices
 2020-08-19 14:51:04 +03:00
Jacek Sieka
a1bd44f4b0
small spec cleanups (#1501) 
* clean up logging a bit
* return error on indexed attestation check
 2020-08-13 13:47:06 +00:00
tersec
81b3c0ea40
update spec refs to v0.12.2 (#1457) 
* update spec refs to v0.12.2 and change a .len.uint64 to .lenu64

* pull back from any non-pure-comment changes, since Jenkins is being wonky
 2020-08-06 13:05:13 +00:00
Dustin Brody
8618cbdf38 keep representations consistent for tests 2020-08-04 23:00:33 +03:00
Dustin Brody
e8c31e4ce8 just in case 2020-08-04 23:00:33 +03:00
Dustin Brody
2c7d207a72 make test passes 2020-08-04 23:00:33 +03:00
Dustin Brody
c142de4b7f be more consistent about pubkeys fed to verify_foo_signature() not being separately initialized, while pubkeys, generally, used for matching purposes, elsewhere explicitly initialized 2020-08-04 23:00:33 +03:00
Dustin Brody
db397cc5f3 lazily initialize validator public keys, which were a third or more of sync CPU usage 2020-08-04 23:00:33 +03:00
tersec
a979568d74
switch mapIt/find to in-place linear search (#1436) 2020-08-04 17:15:27 +00:00
Jacek Sieka
cf8cd8321b
Revert "Lazy crypto [alt design #1369] (#1371)" (#1435) 
This reverts commit 023f7f4518fb8dd9cab2af5faba76b6975893dea.
 2020-08-04 17:15:13 +00:00
tersec
df80071bcf
update attestation and block validation to v0.12.2; clean up getAncestorAt()/get_ancestor() (#1417) 
* update attestation validation to v0.12.2; clean up getAncestorAt()/get_ancestor()

* update beacon block validation to v0.12.2
 2020-08-03 19:47:42 +00:00
tersec
99f2d8e06c
update 14 v0.12.1 spec refs to v0.12.2 (#1400) 2020-07-30 09:47:57 +00:00
Mamy Ratsimbazafy
023f7f4518
Lazy crypto [alt design #1369] (#1371) 
* Lazy loading of crypto objects

* Try to fix incorrect field access by hiding fields but no luck. SSZ/Chronicles/macro bug?

* Fix incorrect blsValue access. was "aggregate" not "chronicles"

* Fix tests that rely on the internal BLSValue representation
 2020-07-29 18:13:05 +00:00
Jacek Sieka
157ddd2ac4
Fork choice fixes 5 (#1381) 
* limit attestations kept in attestation pool

With fork choice updated, the attestation pool only needs to keep track
of attestations that will eventually end up in blocks - we can thus
limit the horizon of attestations that we keep more aggressively.

To get here, we expose getEpochRef which gets metadata about a
particular epochref, and make sure to populate it when a block is added
- this ensures that state rewinds during block addition are minimized.

In addition, we'll use the target root/epoch when validating
attestations - this helps minimize the number of different states that
we need to rewind to, in general.

* remove CandidateChains.justifiedState

unused

* remove BlockPools.Head object

* avoid quadratic quarantine loop

* fix
 2020-07-28 13:54:32 +00:00
tersec
dad3dd5809
update 111 v0.12.1 beacon chain spec refs to v0.12.2 (#1380) 2020-07-27 18:49:46 +02:00
Jacek Sieka
48cebc7157
spec cleanups (#1379) 
* add helper for beacon committee length (used for quickly validating
attestations)
* refactor some attestation checks to do cheap checks early
* validate attestation epoch before computing active validator set
* clean up documentation / comments
* fill state cache on demand
 2020-07-27 16:04:44 +00:00
tersec
20a2525390
v0.12.2 beacon chain protocol update (#1378) 2020-07-27 12:59:57 +02:00
tersec
90708a8287
Prefer converting int` to uint64 and switch foo.len.uint64 to .len64 (#1375) 
* avoid converting from uint64 to int, and where most feasible, int type conversion at all

* .len.uint64 -> .len64

* fix 32-bit compilation

* try keeping state_sim loop variable/bounds as int for 32-bit Azure

* len64 -> lenu64
 2020-07-26 20:55:48 +02:00
Jacek Sieka
e0a18a3105
cache beacon committee size calculation (#1363) 
* cache beacon committee size calculation

this fixes a bug in get_validator_churn_limit as well

* fix

* make committee counts consistently uint64

mixing feels like the worst of the two worlds
 2020-07-23 19:01:07 +02:00
Jacek Sieka
fb2f742972
Fork choice fixes 2 (#1356) 
* fork choice cleanup

* enable v2 pruning
* prefer `get_current_epoch`
* fix finalization check to use correct epoch

* small cleanups

* add `count_active_validators`
* remove misleading logs
* fix justified checkpoint slot calculation in rpc
 2020-07-22 23:01:44 +02:00
tersec
4a9a7be271
faster syncing (#1348) 
* maybe faster syncing

* 80-character lines

* remove instrumentation debugEchos; fix target attestation epoch in attestation pool validation

* use the epoch-granularity matching in attestation.addResolved(...)
 2020-07-22 09:51:45 +02:00
tersec
83abbcb917
drop get_attesting_indices()/get_unslashed_attesting_indices() from 15% to 1% of workload at block_sim at 100k validators (#1351) 2020-07-21 18:35:43 +02:00
Jacek Sieka
8b01284b0e
cache block hash (#1329) 
hash_tree_root was turning up when running beacon_node, turns out to be
repeated hash_tree_root invocations - this pr brings them back down to
normal.

this PR caches the root of a block in the SignedBeaconBlock object -
this has the potential downside that even invalid blocks will be hashed
(as part of deserialization) - later, one could imagine delaying this
until checks have passed

there's also some cleanup of the `cat=` logs which were applied randomly
and haphazardly, and to a large degree are duplicated by other
information in the log statements - in particular, topics fulfill the
same role
 2020-07-16 15:16:51 +02:00
tersec
26e893ffc2
restore EpochRef and flush statecaches on epoch transitions (#1312) 
* restore EpochRef and flush statecaches on epoch transitions

* more targeted cache invalidation

* remove get_empty_per_epoch_cache(); implement simpler but still faster get_beacon_proposer_index()/compute_proposer_index() approach; add some abstraction layer for accessing the shuffled validator indices cache

* reduce integer type conversions

* remove most of rest of integer type conversion in compute_proposer_index()
 2020-07-15 12:44:18 +02:00