status-im/nimbus-eth2
2
0
Fork 0
mirror of https://github.com/status-im/nimbus-eth2.git synced 2025-01-10 06:16:25 +00:00
Code Issues Projects Releases Wiki Activity
2,977 Commits 109 Branches 116 Tags 244 MiB
Commit Graph

713 Commits

Author SHA1 Message Date
Zahary Karadjov
aa6e93a0cd Remove time.humaneStr and associated code 2021-01-26 18:46:33 +02:00
tersec
1bdbf099cc
use IntSet rather than HashSet[ValidatorIndex] (#2267) 
* use IntSet rather than HashSet[ValidatorIndex]

* add bounds check before uint64 -> int conversion

* use intsets in block transitions

* remove superfluous Nim issue explanation/reference
 2021-01-26 12:52:00 +01:00
Mamy Ratsimbazafy
70a03658e3
Block validation flow v2 + Batch (serial) sig verification (#2250) 
* bump nim-blscurve

* Outline the block validation flow

* introduce the SigVerified types, pass the tests

* Split clearance/quarantine to prepare for batch crypto verif

* Add a batch signature collector

* Make clearance use SigVerified block and split verification between crypto and state transition

* Always use signedBeaconBlock for the onBlockAdded callback

* RANDAO signing_root is the epoch instead of the full block

* Support skipping BLS for testing

* Fix compilation of the validator client

* Try to fix strange errors MacOS and Jenkins (Clang, unknown type name br_hmac_drbg_context in stdlib_assertions.nim.c)

* address https://github.com/status-im/nimbus-eth2/pull/2250#discussion_r561819858

* address https://github.com/status-im/nimbus-eth2/pull/2250#discussion_r561828025

* onBlockAdded callback should use TrustedSignedBeaconBlock https://github.com/status-im/nimbus-eth2/pull/2250#discussion_r561837261

* address https://github.com/status-im/nimbus-eth2/pull/2250#discussion_r561828946

* Use the application RNG: https://github.com/status-im/nimbus-eth2/pull/2250#discussion_r561815336

* Improve codegen of conversion zero-cost)

* Quick fixes with loadWithCache after #2259 (TODO: graceful error since pubkey validations is now done first in signatures_batch)

* Graceful handle rogue pubkeys and signatures now that those are lazy-loaded
 2021-01-25 20:45:48 +02:00
Jacek Sieka
5713a3ce4c
performance fixes (#2259) 
* performance fixes

* don't mark tree cache as dirty on read-only List accesses
* store only blob in memory for keys and signatures, parse blob lazily
* compare public keys by blob instead of parsing / converting to raw
* compare Eth2Digest using non-constant-time comparison
* avoid some unnecessary validator copying

This branch will in particular speed up deposit processing which has
been slowing down block replay.

Pre (mainnet, 1600 blocks):

```
All time are ms
     Average,       StdDev,          Min,          Max,      Samples,         Test
Validation is turned off meaning that no BLS operations are performed
    3450.269,        0.000,     3450.269,     3450.269,            1, Initialize DB
       0.417,        0.822,        0.036,       21.098,         1400, Load block from database
      16.521,        0.000,       16.521,       16.521,            1, Load state from database
      27.906,       50.846,        8.104,     1507.633,         1350, Apply block
      52.617,       37.029,       20.640,      135.938,           50, Apply epoch block
```

Post:

```
    3502.715,        0.000,     3502.715,     3502.715,            1, Initialize DB
       0.080,        0.560,        0.035,       21.015,         1400, Load block from database
      17.595,        0.000,       17.595,       17.595,            1, Load state from database
      15.706,       11.028,        8.300,      107.537,         1350, Apply block
      33.217,       12.622,       17.331,       60.580,           50, Apply epoch block
```

* more perf fixes

* load EpochRef cache into StateCache more aggressively
* point out security concern with public key cache
* reuse proposer index from state when processing block
* avoid genericAssign in a few more places
* don't parse key when signature is unparseable
* fix `==` overload for Eth2Digest
* preallocate validator list when getting active validators
* speed up proposer index calculation a little bit
* reuse cache when replaying blocks in ncli_db
* avoid a few more copying loops

```
     Average,       StdDev,          Min,          Max,      Samples,         Test
Validation is turned off meaning that no BLS operations are performed
    3279.158,        0.000,     3279.158,     3279.158,            1, Initialize DB
       0.072,        0.357,        0.035,       13.400,         1400, Load block from database
      17.295,        0.000,       17.295,       17.295,            1, Load state from database
       5.918,        9.896,        0.198,       98.028,         1350, Apply block
      15.888,       10.951,        7.902,       39.535,           50, Apply epoch block
       0.000,        0.000,        0.000,        0.000,            0, Database block store
```

* clear full balance cache before processing rewards and penalties

```
All time are ms
     Average,       StdDev,          Min,          Max,      Samples,         Test
Validation is turned off meaning that no BLS operations are performed
    3947.901,        0.000,     3947.901,     3947.901,            1, Initialize DB
       0.124,        0.506,        0.026,      202.370,       363345, Load block from database
      97.614,        0.000,       97.614,       97.614,            1, Load state from database
       0.186,        0.188,        0.012,       99.561,       357262, Advance slot, non-epoch
      14.161,        5.966,        1.099,      395.511,        11524, Advance slot, epoch
       1.372,        4.170,        0.017,      276.401,       363345, Apply block, no slot processing
       0.000,        0.000,        0.000,        0.000,            0, Database block store
```
 2021-01-25 13:04:18 +01:00
tersec
7f5a26002d
remove some skipBlsValidation usage/handling in tests (#2258) 2021-01-22 14:29:04 +01:00
Zahary Karadjov
7571e74dbd Fix #2252 (consistent typing for the 'delay' log property) 2021-01-22 15:09:21 +02:00
Dustin Brody
a16f5afcd5 pre-emptive duplicate validator detection heuristic 2021-01-21 16:03:02 +02:00
tersec
921fe5a68f
initial infrastructure for state diffs (#2087) 
Initial infrastructure for state diffs
 2021-01-18 22:34:41 +02:00
Zahary Karadjov
338428cbd7 Add Eth1 deposits simulation to block_sim 2021-01-04 13:22:00 +02:00
tersec
97c4f7c5c0
fix subnet calculation in RPC and insert broadcast attestations into node's pool (#2207) 
* fix subnet calculation in RPC and insert broadcast attestations into node's pool

* unify codepaths to ensure only mostly-checked-to-be-valid attestations enter the pool, even from node's own broadcasts

* update attestation pool tests for new validateAttestation param
 2020-12-23 13:59:04 +01:00
Jacek Sieka
7d5edb4353
use new stew helpers for assignment (#2172) 
* bump libp2p (reduces libp2p gossip memory usage to ~1/3)
* use "generic" assign version
 2020-12-16 09:37:22 +01:00
tersec
8d1443f03c
detect already-aggregate-voted condition before attestation pool; add is_aggregator tests (#2170) 
* detect already-aggregate-voted condition before attestation pool; add is_aggregator tests

* replace pair of attestation-per-epoch tracking lists with single list and remove Option use

* fix attestation condition

* use safer type conversions; add more is_aggregator tests
 2020-12-14 20:58:32 +00:00
Jacek Sieka
dbcc0686ff delay pruning of cache for finalized epoch (fixes #2049) 2020-11-20 20:57:50 +02:00
tersec
3c4cf95d0e
remove v0.12.3 and medalla support (#2038) 2020-11-20 14:49:49 +01:00
tersec
1d7fb2ed0c
remove {.inline.} pragmas (#2033) 
* remove {.inline.} pragmas

* re-add inline on bitseqs functions and tweak inlining threshold

* remove macOS/LLVM inlining setting; revert non-init/module-local/tests inline pragma removal
 2020-11-20 11:00:22 +01:00
tersec
21c4ce8fd4
remove superfluous TODOs/not-really-TODOs, type conversion, imports (#2025) 2020-11-16 17:10:51 +01:00
Zahary Karadjov
b022dc4d1f Use O(n) algorithm in initialize_beacon_state_from_eth1; Avoid unnecessary merkle proofs generation 2020-11-15 21:40:40 +02:00
tersec
602f344525
remove v0.12.3 EF test vectors to save 2.7GB of space in CI instances (#2017) 2020-11-13 22:16:56 +01:00
Zahary Karadjov
5e45e7429e Rewrite all HTTP(S) web3 URLs to WebSocket URls 2020-11-12 22:29:43 +02:00
tersec
b9e122a60c
address some TODOs (#2005) 2020-11-12 19:24:07 +00:00
Zahary Karadjov
389c11743a
Review TODO items and self-assign the most important ones 2020-11-10 20:41:04 +02:00
tersec
a010ef2061
update tests from v1.0.0-rc.0 to v1.0.0 (#1976) 2020-11-09 17:44:18 +00:00
tersec
271df8b604
bump 1.0.0rc-0 spec refs to 1.0.0 (#1974) 2020-11-09 14:18:55 +00:00
Zahary Karadjov
e9b9cd75ee Rename binaries; Mimic the original repo layout in the distribution 2020-11-09 11:38:52 +02:00
Zahary Karadjov
a8a66fe3f6 Fix #1784; Handle Infura HTTPS URLs 2020-11-06 11:15:32 +02:00
Jacek Sieka
95f5f76180
Datatype cleanup (#1953) 
* clear up spec todo

* test fix

* remove unnecessary toSszType

* type

* one more
 2020-11-04 21:52:47 +00:00
Jacek Sieka
fc7885b27e Store block summary in database 
This introcudes a cache for block summaries, useful for instantiating
the block dag on startup, bringing medalla startup times down from
minutes to seconds.

This is something of a temporary band-aid that would be obsoleted by a
finalized block store.
 2020-11-04 11:28:55 +02:00
Zahary Karadjov
6ee80651c7 Rename mainchain_monitor to eth1_monitor 2020-11-03 23:23:10 +02:00
Zahary Karadjov
2f29e3e7f3 Further reduce the number of RPC requests; Significantly faster ETH1 syncing 2020-11-03 23:23:10 +02:00
Zahary Karadjov
e414d964f0 Address review comments: delete obsolete TODO item 2020-11-03 23:23:10 +02:00
Zahary Karadjov
14b2d4324d openarray -> openArray 2020-11-03 23:23:10 +02:00
Zahary Karadjov
79f1b6117e Remove dead code from the beacon node binary; Fix 'make test' 2020-11-03 23:23:10 +02:00
Zahary Karadjov
07206ca264 An efficient algorithm for incremental generation of multiple merkle proofs 2020-11-03 23:23:10 +02:00
Zahary Karadjov
67113373f9 Work around BLST build issues 2020-10-30 23:56:04 +02:00
Zahary Karadjov
f76679810b Fix #1855; Add support for wallet recovery passwords 2020-10-30 23:56:04 +02:00
Eugene Kabanov
eee01a32d5
Regression fix of eth2_network_simulation on Windows. (#1900) 
* Concentrate all sensitive writeFile/createPath calls in one place.
Fix eth2_network_simulation for Windows.

* Remove artifacts.

* fix import

Co-authored-by: Jacek Sieka <jacek@status.im>
 2020-10-27 12:04:17 +01:00
Jacek Sieka
b246d51ac1 test fix 2020-10-23 19:23:36 +03:00
Jacek Sieka
7c0b4d28d2 speed up reward/penalty calculation 
Calculating rewards/penalties is slow due to how we compute sets of
attestations validators then use the sets for inclusion checks, to see
who attested. The dominant function during validated block processing /
epoch processing is hash set building and lookup.

This PR inverts the flow by removing the sets and creating a single
large validator status list, then applying all relevant state
attestations, then updating rewards and penalties.

This provides a 10x speedup to epoch processing which in turn speeds up
both empty slot and block processing - for example, on startup, we
replay all non-finalized blocks to prime fork choice - the same when
validating attestations or replaying states on reorg.
 2020-10-23 19:23:36 +03:00
tersec
a136c2e95a
bump libp2p; integrate pubsub.ValidationResult into extended validation (#1893) 2020-10-20 12:31:20 +00:00
tersec
e87e0ef179
update to 1.0.0-rc.0 message ID; unexport funcs; enable test in mainnet; remove unused subscribe-with-nondummy-handler function; fix gossip received metric (#1884) 2020-10-20 08:54:11 +00:00
tersec
c446eb7cff
update v1.0.0-rc.0 preset spec references (#1891) 
* update ve1.0.0-rc.0 preset spec references

* remove runtime preset ETH1_FOLLOW_DISTANCE from preset files; remove two CI build items to try to keep Travis from timing out
 2020-10-19 13:53:23 +00:00
Jacek Sieka
df43b8aa8b
save some more states after all (#1887) 
Don't save states when replaying history, but do save states when
applying new blocks (!)
 2020-10-18 15:47:39 +00:00
Zahary Karadjov
948d15ea95 Follow-up on #1687 Don't ask for password on invalid keystores 2020-10-16 11:45:39 +03:00
Zahary Karadjov
5f6bdc6709 Store all deposit-derived data in memory 2020-10-15 20:15:51 +03:00
Zahary Karadjov
7a577b2cef More tests for getBlockRange 2020-10-15 20:15:51 +03:00
Zahary Karadjov
2152dc6136 Simplify the mainchain monitor 2020-10-15 20:15:51 +03:00
Zahary Karadjov
4d66914f5a Fix the test suite 2020-10-15 20:15:51 +03:00
Zahary Karadjov
02a3770803 Adds a fuzzing test for the validator client CLI 
Also removes an obsolete way to add validators through the command line
 2020-10-15 20:15:51 +03:00
Zahary Karadjov
8ce0fc3a89 Address #1687 
This addresses the issues by detecting and rejecting keystores with
incorrect PBKDF2 and SCrypt params. It also bumps the version of
nim-json-serialization to include a bugfix for incorrect parsing
of json files featuring comments.
 2020-10-15 20:15:51 +03:00
Jacek Sieka
6b9419e547
fix db growth on attestation processing (#1860) 
It turns out that we often save lots of states in the database that are
the result of empty slot processing only - here, we make sure to only
save a state if a block follows - this fixes several issues:

* empty slot states are not always pruned leading to state database size
explosion
* storing states is (very) slow which slows down processing in general,
so we should only do it when it's likely to be useful
* attestation processing doesn't get stuck on saving random states that
won't appear in the chain history
 2020-10-15 14:28:44 +02:00