status-im/nimbus-eth2
2
0
Fork 0
mirror of https://github.com/status-im/nimbus-eth2.git synced 2025-01-12 23:34:44 +00:00
Code Issues Projects Releases Wiki Activity
3,016 Commits 106 Branches 116 Tags 245 MiB
Commit Graph

115 Commits

Author SHA1 Message Date
Mamy Ratsimbazafy
03f47c8f2f
Slashing protection refactor - EIP 3076 (#2094) 
* Create CLI tool for slashing export

* Use SQLite as a DB instead of a KV-store

* Keeps v1 and v2 DBs around

* Uses the same schema as Lighthouse v1.1.0

* Passes all interchange tests + skeleton of finalization pruning

* Removes tests that would violate v5 / minimal slashing DB and MinSlot rules

* Migration tool added using low-watermark scheme for faster migration of large number of validators
 2021-02-09 17:23:06 +02:00
Dustin Brody
67e4a045a3 simplify doppelganger detection to boolean 2021-02-03 20:55:33 +02:00
tersec
6141286547
rename doppelganger protection to doppelganger detection; switch default from warn to stop (#2281) 2021-02-01 12:18:16 +01:00
Zahary Karadjov
fa99c3b417 Fix #2261 
Also bumps Confutils to allow setting the hidden --web3-mode param
(to allow testing the eth1 syncing without validators)
 2021-01-30 01:32:20 +02:00
Dustin Brody
f7e327234f notice is too spammy for a situation-nominal message 2021-01-30 00:17:54 +02:00
Dustin Brody
281853eee8 rename options and internal structures to doppelgangerFoo and remove probing 2021-01-30 00:17:54 +02:00
Zahary Karadjov
aa6e93a0cd Remove time.humaneStr and associated code 2021-01-26 18:46:33 +02:00
Mamy Ratsimbazafy
70a03658e3
Block validation flow v2 + Batch (serial) sig verification (#2250) 
* bump nim-blscurve

* Outline the block validation flow

* introduce the SigVerified types, pass the tests

* Split clearance/quarantine to prepare for batch crypto verif

* Add a batch signature collector

* Make clearance use SigVerified block and split verification between crypto and state transition

* Always use signedBeaconBlock for the onBlockAdded callback

* RANDAO signing_root is the epoch instead of the full block

* Support skipping BLS for testing

* Fix compilation of the validator client

* Try to fix strange errors MacOS and Jenkins (Clang, unknown type name br_hmac_drbg_context in stdlib_assertions.nim.c)

* address https://github.com/status-im/nimbus-eth2/pull/2250#discussion_r561819858

* address https://github.com/status-im/nimbus-eth2/pull/2250#discussion_r561828025

* onBlockAdded callback should use TrustedSignedBeaconBlock https://github.com/status-im/nimbus-eth2/pull/2250#discussion_r561837261

* address https://github.com/status-im/nimbus-eth2/pull/2250#discussion_r561828946

* Use the application RNG: https://github.com/status-im/nimbus-eth2/pull/2250#discussion_r561815336

* Improve codegen of conversion zero-cost)

* Quick fixes with loadWithCache after #2259 (TODO: graceful error since pubkey validations is now done first in signatures_batch)

* Graceful handle rogue pubkeys and signatures now that those are lazy-loaded
 2021-01-25 20:45:48 +02:00
tersec
8c48d44788
remove superfluous/overly aggressive, timing-dependent assertion (#2264) 2021-01-25 17:52:55 +01:00
Jacek Sieka
5713a3ce4c
performance fixes (#2259) 
* performance fixes

* don't mark tree cache as dirty on read-only List accesses
* store only blob in memory for keys and signatures, parse blob lazily
* compare public keys by blob instead of parsing / converting to raw
* compare Eth2Digest using non-constant-time comparison
* avoid some unnecessary validator copying

This branch will in particular speed up deposit processing which has
been slowing down block replay.

Pre (mainnet, 1600 blocks):

```
All time are ms
     Average,       StdDev,          Min,          Max,      Samples,         Test
Validation is turned off meaning that no BLS operations are performed
    3450.269,        0.000,     3450.269,     3450.269,            1, Initialize DB
       0.417,        0.822,        0.036,       21.098,         1400, Load block from database
      16.521,        0.000,       16.521,       16.521,            1, Load state from database
      27.906,       50.846,        8.104,     1507.633,         1350, Apply block
      52.617,       37.029,       20.640,      135.938,           50, Apply epoch block
```

Post:

```
    3502.715,        0.000,     3502.715,     3502.715,            1, Initialize DB
       0.080,        0.560,        0.035,       21.015,         1400, Load block from database
      17.595,        0.000,       17.595,       17.595,            1, Load state from database
      15.706,       11.028,        8.300,      107.537,         1350, Apply block
      33.217,       12.622,       17.331,       60.580,           50, Apply epoch block
```

* more perf fixes

* load EpochRef cache into StateCache more aggressively
* point out security concern with public key cache
* reuse proposer index from state when processing block
* avoid genericAssign in a few more places
* don't parse key when signature is unparseable
* fix `==` overload for Eth2Digest
* preallocate validator list when getting active validators
* speed up proposer index calculation a little bit
* reuse cache when replaying blocks in ncli_db
* avoid a few more copying loops

```
     Average,       StdDev,          Min,          Max,      Samples,         Test
Validation is turned off meaning that no BLS operations are performed
    3279.158,        0.000,     3279.158,     3279.158,            1, Initialize DB
       0.072,        0.357,        0.035,       13.400,         1400, Load block from database
      17.295,        0.000,       17.295,       17.295,            1, Load state from database
       5.918,        9.896,        0.198,       98.028,         1350, Apply block
      15.888,       10.951,        7.902,       39.535,           50, Apply epoch block
       0.000,        0.000,        0.000,        0.000,            0, Database block store
```

* clear full balance cache before processing rewards and penalties

```
All time are ms
     Average,       StdDev,          Min,          Max,      Samples,         Test
Validation is turned off meaning that no BLS operations are performed
    3947.901,        0.000,     3947.901,     3947.901,            1, Initialize DB
       0.124,        0.506,        0.026,      202.370,       363345, Load block from database
      97.614,        0.000,       97.614,       97.614,            1, Load state from database
       0.186,        0.188,        0.012,       99.561,       357262, Advance slot, non-epoch
      14.161,        5.966,        1.099,      395.511,        11524, Advance slot, epoch
       1.372,        4.170,        0.017,      276.401,       363345, Apply block, no slot processing
       0.000,        0.000,        0.000,        0.000,            0, Database block store
```
 2021-01-25 13:04:18 +01:00
Zahary Karadjov
7571e74dbd Fix #2252 (consistent typing for the 'delay' log property) 2021-01-22 15:09:21 +02:00
Dustin Brody
a16f5afcd5 pre-emptive duplicate validator detection heuristic 2021-01-21 16:03:02 +02:00
tersec
97c4f7c5c0
fix subnet calculation in RPC and insert broadcast attestations into node's pool (#2207) 
* fix subnet calculation in RPC and insert broadcast attestations into node's pool

* unify codepaths to ensure only mostly-checked-to-be-valid attestations enter the pool, even from node's own broadcasts

* update attestation pool tests for new validateAttestation param
 2020-12-23 13:59:04 +01:00
Jacek Sieka
de779c7812 update validator metrics on startup 2020-12-16 19:42:19 +02:00
Jacek Sieka
7d5edb4353
use new stew helpers for assignment (#2172) 
* bump libp2p (reduces libp2p gossip memory usage to ~1/3)
* use "generic" assign version
 2020-12-16 09:37:22 +01:00
tersec
8d1443f03c
detect already-aggregate-voted condition before attestation pool; add is_aggregator tests (#2170) 
* detect already-aggregate-voted condition before attestation pool; add is_aggregator tests

* replace pair of attestation-per-epoch tracking lists with single list and remove Option use

* fix attestation condition

* use safer type conversions; add more is_aggregator tests
 2020-12-14 20:58:32 +00:00
tersec
72d00e33b8
always wait for 2/3 into slot for aggregate attestations (#2158) 2020-12-07 20:32:34 +01:00
tersec
26ea76bbdf
don't lag aggregated attestations by a slot (#2150) 
* don't lag aggregated attestations by a slot

* don't use aggregation topic at all

* use aggregates again, but with aggressively low ATTESTATION_PROPAGATION_SLOT_RANGE; seems to hold on to LH 1.0 nodes

* clean up scaffolding and double ATTESTATION_PROPAGATION_SLOT_RANGE to 16

* increase ATTESTATION_PROPAGATION_SLOT_RANGE to 24

* increase ATTESTATION_PROPAGATION_SLOT_RANGE to 28 and isolate in only used function due to customization; remove TRAILING_DISTANCE machinery
 2020-12-07 12:56:49 +01:00
cheatfate
7e4ff7a740 Implementation of post_v1_beacon_pool_attester_slashings() and post_v1_beacon_pool_proposer_slashings(). 
Remove doAssert.
 2020-12-03 11:40:08 +02:00
Zahary Karadjov
4eaf29875a Less risk for Eth1-induced delays in block proposal 2020-12-01 15:38:50 +02:00
Zahary Karadjov
ac9bdde543
Don't rely on a metric value for the ETH display in the status bar 2020-11-29 23:35:39 +02:00
Zahary Karadjov
3c0dfc2fbe Implement the 'deposits exit' command; Remove 'deposits create' 2020-11-29 23:08:07 +02:00
Zahary Karadjov
bf2673abc4
Restore the ETH display in the status bar 2020-11-28 20:53:51 +02:00
Jacek Sieka
8685eb7042 add validator balance metrics 2020-11-28 16:22:09 +02:00
zah
372c9b798c
Fix the corrupted database state on Pyrmont nodes; Add mainnet genesis (#2056) 
* Handle some web3 timeouts better

* Add support for developer .env files

* Eth1 improvements; Mainnet genesis state

Notable changes:

* The deposits table have been removed from the database. The client
  will no longer process all deposits on start-up.

* The network metadata now includes a "state snapshot" of the deposit
  contract. This allows the client to skip syncing deposits made prior
  to the snapshot (i.e. genesis). Suitable metadata added for Pyrmont
  and Mainnet.

* The Eth1 monitor won't be started unless there are validators attached
  to the node.

* The genesis detection code is now optional and disabled by default

* Bugfix: The client should not produce blocks that will fail validation
  when it hasn't downloaded the latest deposits yet

* Bugfix: Work around the database corruption affecting Pyrmont nodes

* Remove metadata for Toledo and Medalla
 2020-11-24 22:21:47 +01:00
Jacek Sieka
dbcc0686ff delay pruning of cache for finalized epoch (fixes #2049) 2020-11-20 20:57:50 +02:00
Zahary Karadjov
e22248bca1 Tentative fix for the Pyrmont Eth1 voting issue 2020-11-20 17:13:51 +02:00
tersec
9e716b32bd
address some XDeclaredButNotUsed hints (#2028) 2020-11-17 11:14:53 +01:00
Jacek Sieka
a6b188bfd4
misc fixes (#2027) 
* log when database is loading (to avoid confusion)
* generate network keys later during startup
* fix quarantine not scheduling chain of parents for download and
increase size to one epoch
* log validator count, enr and peerid more clearly on startup
 2020-11-16 20:15:43 +01:00
tersec
e2f161dbf7
fix attestation sending schedules to avoid timing attack (#1995) 2020-11-16 10:44:18 +01:00
cheatfate
87c92ea05f Address review comments. 
Remove `-8.0` bucket because `-4.0` is a limit.
 2020-11-12 21:52:01 +02:00
cheatfate
dbb058682c Add beacon_attestation_sent_delay metric. 2020-11-12 21:52:01 +02:00
Zahary Karadjov
389c11743a
Review TODO items and self-assign the most important ones 2020-11-10 20:41:04 +02:00
tersec
271df8b604
bump 1.0.0rc-0 spec refs to 1.0.0 (#1974) 2020-11-09 14:18:55 +00:00
Jacek Sieka
95f5f76180
Datatype cleanup (#1953) 
* clear up spec todo

* test fix

* remove unnecessary toSszType

* type

* one more
 2020-11-04 21:52:47 +00:00
Zahary Karadjov
6ee80651c7 Rename mainchain_monitor to eth1_monitor 2020-11-03 23:23:10 +02:00
Zahary Karadjov
2f29e3e7f3 Further reduce the number of RPC requests; Significantly faster ETH1 syncing 2020-11-03 23:23:10 +02:00
Zahary Karadjov
18639c3eff Don't require requests that might fail on non-archive Geth nodes 2020-11-03 23:23:10 +02:00
Sacha Saint-Leger
7294d98ed1
Update validator_duties.nim (Improve node out of sync message) (#1933) 
* Update validator_duties.nim

Improve node out of sync message

* Update validator_duties.nim
 2020-11-02 08:52:16 +01:00
Jacek Sieka
ff3f8741de
handle fork choice selecting a head unknown to chaindag (#1907) 
* move head slot metric to head selection
* avoid redundant head selection in validator duties
 2020-10-28 07:55:36 +00:00
Jacek Sieka
499e5ca991
misc memory and perf fixes (#1899) 
* misc memory and perf fixes

* use EpochRef for attestation aggregation
* compress effective balances in memory (medalla unfinalized: 4gb ->
1gb)
* avoid hitting db when rewinding to head or clearance state
* avoid hitting db when blocks can be applied to in-memory state -
speeds up startup considerably
* avoid storing epochref in fork choice
* simplify and speed up beacon block creation flow - avoids state reload
thanks to head rewind optimization
* iterator-based committee and attestation participation help avoid lots
of small memory allocations throughout epoch transition (40% speedup on
epoch processing, for example during startup)

* add constant for threshold
 2020-10-22 12:53:33 +02:00
cheatfate
1742a82ca7 Proper fix for writeFile() places. 
Comment unused/insecure procedure.
 2020-10-15 20:18:33 +03:00
Zahary Karadjov
e6320e5881 Address #1584 Don't keep all deposits in memory (persist them to disk) 2020-10-15 20:15:51 +03:00
tersec
3ee2dd8da4
p2p-interface spec ref bump (except non-updated places) (#1862) 2020-10-12 14:37:14 +00:00
tersec
1994ffe5a0
update 130+ spec references from v0.12.3 to v1.0.0-rc1 (#1854) 2020-10-12 08:59:24 +00:00
tersec
f08f44b9a2
in exit pool, bundle receive messages into beaconblocks (#1812) 
* in exit pool, filter out already-packaged messages; bundle remaining messages into beaconblocks

* filter messages at block construction time

* allow adding up to intended capacity of buffers, beyond per-block limits

* document rationale/design for filtering mechanism
 2020-10-07 16:57:21 +00:00
Mamy Ratsimbazafy
c92d228ec8
Fix non-split attesting protection (#1804) 
* address #1679 - att protection in non-split mode

* AttachedValidator public_key -> pubkey

* Always build with slashing protection

* Remove now unnecessary forwarding impl proc
 2020-10-06 10:51:33 +02:00
cheatfate
82228fe471 Fix last createDir. 2020-10-05 22:19:50 +03:00
Mamy Ratsimbazafy
0280d6c73e
Revisiting log levels (#1788) 
* Update log level - https://github.com/status-im/nim-beacon-chain/issues/1779 https://github.com/status-im/nim-beacon-chain/issues/1785

* Address review comments

* Document the logging strategy [skip ci]
 2020-10-01 20:56:42 +02:00
tersec
f96ad87d28
switch another 50+ spec refs from v0.12.2 to v0.12.3 (#1749) 2020-09-25 11:52:50 +00:00