status-im/nimbus-eth2
2
0
Fork 0
mirror of https://github.com/status-im/nimbus-eth2.git synced 2025-01-11 06:46:10 +00:00
Code Issues Projects Releases Wiki Activity
3,304 Commits 108 Branches 116 Tags 245 MiB
Commit Graph

1904 Commits

Author SHA1 Message Date
Dmitriy Ryajov
64e4d74f87
pass max peers to libp2p (#2265) 2021-01-26 10:35:22 +01:00
Mamy Ratsimbazafy
70a03658e3
Block validation flow v2 + Batch (serial) sig verification (#2250) 
* bump nim-blscurve

* Outline the block validation flow

* introduce the SigVerified types, pass the tests

* Split clearance/quarantine to prepare for batch crypto verif

* Add a batch signature collector

* Make clearance use SigVerified block and split verification between crypto and state transition

* Always use signedBeaconBlock for the onBlockAdded callback

* RANDAO signing_root is the epoch instead of the full block

* Support skipping BLS for testing

* Fix compilation of the validator client

* Try to fix strange errors MacOS and Jenkins (Clang, unknown type name br_hmac_drbg_context in stdlib_assertions.nim.c)

* address https://github.com/status-im/nimbus-eth2/pull/2250#discussion_r561819858

* address https://github.com/status-im/nimbus-eth2/pull/2250#discussion_r561828025

* onBlockAdded callback should use TrustedSignedBeaconBlock https://github.com/status-im/nimbus-eth2/pull/2250#discussion_r561837261

* address https://github.com/status-im/nimbus-eth2/pull/2250#discussion_r561828946

* Use the application RNG: https://github.com/status-im/nimbus-eth2/pull/2250#discussion_r561815336

* Improve codegen of conversion zero-cost)

* Quick fixes with loadWithCache after #2259 (TODO: graceful error since pubkey validations is now done first in signatures_batch)

* Graceful handle rogue pubkeys and signatures now that those are lazy-loaded
 2021-01-25 20:45:48 +02:00
Jacek Sieka
5ca10d3de2 avoid copying keys for trusted signatures 
this removes `is_valid_indexed_attestion` from performance benchmarks
when replaying blocks
 2021-01-25 19:48:02 +02:00
Jacek Sieka
8f6ce4f88d shorten idle timeout 
this introduces the change more gently, to avoid adverse effects - the
timeout value can subsequently be tuned based on further
experimentation.
 2021-01-25 19:44:05 +02:00
tersec
7d74d3bfbc
only subscribe to subnets when aggregating (#2254) 
Only subscribe to subnets when aggregating
 2021-01-25 19:39:56 +02:00
tersec
8c48d44788
remove superfluous/overly aggressive, timing-dependent assertion (#2264) 2021-01-25 17:52:55 +01:00
Jacek Sieka
5713a3ce4c
performance fixes (#2259) 
* performance fixes

* don't mark tree cache as dirty on read-only List accesses
* store only blob in memory for keys and signatures, parse blob lazily
* compare public keys by blob instead of parsing / converting to raw
* compare Eth2Digest using non-constant-time comparison
* avoid some unnecessary validator copying

This branch will in particular speed up deposit processing which has
been slowing down block replay.

Pre (mainnet, 1600 blocks):

```
All time are ms
     Average,       StdDev,          Min,          Max,      Samples,         Test
Validation is turned off meaning that no BLS operations are performed
    3450.269,        0.000,     3450.269,     3450.269,            1, Initialize DB
       0.417,        0.822,        0.036,       21.098,         1400, Load block from database
      16.521,        0.000,       16.521,       16.521,            1, Load state from database
      27.906,       50.846,        8.104,     1507.633,         1350, Apply block
      52.617,       37.029,       20.640,      135.938,           50, Apply epoch block
```

Post:

```
    3502.715,        0.000,     3502.715,     3502.715,            1, Initialize DB
       0.080,        0.560,        0.035,       21.015,         1400, Load block from database
      17.595,        0.000,       17.595,       17.595,            1, Load state from database
      15.706,       11.028,        8.300,      107.537,         1350, Apply block
      33.217,       12.622,       17.331,       60.580,           50, Apply epoch block
```

* more perf fixes

* load EpochRef cache into StateCache more aggressively
* point out security concern with public key cache
* reuse proposer index from state when processing block
* avoid genericAssign in a few more places
* don't parse key when signature is unparseable
* fix `==` overload for Eth2Digest
* preallocate validator list when getting active validators
* speed up proposer index calculation a little bit
* reuse cache when replaying blocks in ncli_db
* avoid a few more copying loops

```
     Average,       StdDev,          Min,          Max,      Samples,         Test
Validation is turned off meaning that no BLS operations are performed
    3279.158,        0.000,     3279.158,     3279.158,            1, Initialize DB
       0.072,        0.357,        0.035,       13.400,         1400, Load block from database
      17.295,        0.000,       17.295,       17.295,            1, Load state from database
       5.918,        9.896,        0.198,       98.028,         1350, Apply block
      15.888,       10.951,        7.902,       39.535,           50, Apply epoch block
       0.000,        0.000,        0.000,        0.000,            0, Database block store
```

* clear full balance cache before processing rewards and penalties

```
All time are ms
     Average,       StdDev,          Min,          Max,      Samples,         Test
Validation is turned off meaning that no BLS operations are performed
    3947.901,        0.000,     3947.901,     3947.901,            1, Initialize DB
       0.124,        0.506,        0.026,      202.370,       363345, Load block from database
      97.614,        0.000,       97.614,       97.614,            1, Load state from database
       0.186,        0.188,        0.012,       99.561,       357262, Advance slot, non-epoch
      14.161,        5.966,        1.099,      395.511,        11524, Advance slot, epoch
       1.372,        4.170,        0.017,      276.401,       363345, Apply block, no slot processing
       0.000,        0.000,        0.000,        0.000,            0, Database block store
```
 2021-01-25 13:04:18 +01:00
Zahary Karadjov
7571e74dbd Fix #2252 (consistent typing for the 'delay' log property) 2021-01-22 15:09:21 +02:00
Zahary Karadjov
960666d1ed
Remove std/random again 2021-01-21 19:39:04 +02:00
Mamy Ratsimbazafy
718feef802
Fix unstable after #2244 (#2255) 2021-01-21 18:27:24 +01:00
cheatfate
69d6ffd990 Remove watchTask(). 2021-01-21 16:22:01 +02:00
Dustin Brody
a16f5afcd5 pre-emptive duplicate validator detection heuristic 2021-01-21 16:03:02 +02:00
tersec
55ecb61c3a cycle attestation subnets every slot (#2240) 
Cycle attestation subnets every slot
 2021-01-19 19:44:03 +02:00
tersec
921fe5a68f
initial infrastructure for state diffs (#2087) 
Initial infrastructure for state diffs
 2021-01-18 22:34:41 +02:00
Jacek Sieka
c0e11cfba6 shorten idle timeout 2021-01-18 22:27:34 +02:00
Jacek Sieka
76f15302a7 better timesharing in eth2 processor 
* use `idleAsync` to more evenly divide cpu attention when syncing in
particular - this gives networking better latency
* more strict exception handling in eth2_processor
 2021-01-18 22:27:34 +02:00
Kim De Mey
6fabefa76f
Fix discv5 loop in case of no nodes (#2243) 
* Fix discovery loop in case of no peers in routing table

* local testnet: Stop searching when amount of peers in testnet is reached
 2021-01-18 14:13:26 +01:00
tersec
0fce8ad0d7
Revert "only checkpoint every four slots (#2236)" (#2242) 
This reverts commit 7da16f49086fc21ce10766eb0cc00decc9d0fd81.
 2021-01-18 11:02:56 +01:00
Zahary Karadjov
c8c819359c
More clear error message when a validator exit was rejected 2021-01-15 19:40:05 +02:00
tersec
7da16f4908
only checkpoint every four slots (#2236) 
* only checkpoint every four slots

* only checkpoint every 16 slots

* every 8 slots

* every 4 slots; 8 seems probably okay, but be a bit conservative
 2021-01-15 05:23:54 +00:00
Giovanni Petrantoni
295e3c9c73
Topics validation and direct peers (#2237) 
* pick the right libp2p branch

* add topics validation
 2021-01-15 04:17:06 +00:00
tersec
fa75c477cd
only initially subscribe to relevant attestation subnets (#2231) 2021-01-14 09:43:21 +01:00
Kim De Mey
66d8f317cd
Use async queryRandom instead of inefficient randomNodes to discover nodes (#2211) 2021-01-14 08:58:13 +01:00
tersec
0fad1b6b26
don't special-case zero-validator subnet cycling (#2230) 2021-01-12 17:17:43 +01:00
tersec
dde973e2d4
allow always-on subscription to all attestation subnets when gossiping (#2225) 
* allow always-on subscription to all attestation subnets when gossiping

* in subscribe-all-subnets mode, consider all subnets to be stability subnets for ENR purposes
 2021-01-12 13:43:15 +01:00
Giovanni Petrantoni
a3a651b565
always enable topic and aggreate metric topics (#2229) 2021-01-12 04:27:09 +01:00
Kim De Mey
b99a6f3dfd
Warn on Eth1 chain not synced (#2221) 2021-01-11 10:23:09 +01:00
Zahary Karadjov
87955f2d37
v1.0.6 2021-01-10 12:23:07 +02:00
Zahary Karadjov
d803f008c0
v1.0.5 (corrected) 2021-01-10 12:02:55 +02:00
tersec
bbdd16f81c
increase attestation pool lookback window (#2215) 2021-01-07 14:52:02 +01:00
Zahary Karadjov
338428cbd7 Add Eth1 deposits simulation to block_sim 2021-01-04 13:22:00 +02:00
Giovanni Petrantoni
ed24f60f70
remove async from sub/unsub (#2197) 
* remove await/async from sub/unsub

* fix unsubscribe wrong key (missed _snappy)

* use the right libp2p commit hash

* remove unused async

* fix inspector

* fix subnet calculation in RPC and insert broadcast attestations into node's pool

* unify codepaths to ensure only mostly-checked-to-be-valid attestations enter the pool, even from node's own broadcasts

* update attestation pool tests for new validateAttestation param

Co-authored-by: Dustin Brody <tersec@users.noreply.github.com>
 2020-12-24 09:48:52 +01:00
tersec
97c4f7c5c0
fix subnet calculation in RPC and insert broadcast attestations into node's pool (#2207) 
* fix subnet calculation in RPC and insert broadcast attestations into node's pool

* unify codepaths to ensure only mostly-checked-to-be-valid attestations enter the pool, even from node's own broadcasts

* update attestation pool tests for new validateAttestation param
 2020-12-23 13:59:04 +01:00
tersec
afbaa36ef7
make subnet cycling more robust; use one stability subnet/validator; explicitly represent gossip enabled/disabled (#2201) 
* make subnet cycling more robust; use one stability subnet/validator; explicitly represent gossip enabled/disabled

* fix asymmetry in _snappy being used for subscriptions but not unsubscriptions

* remove redundant comment

* minimal RPC and VC support for infoming BN of subnets

* create and verify slot signatures in RPC interface and VC

* loosen old slot check

* because Slot + uint64 works but uint64 + Slot doesn't

* document assumptions for head state use; don't clear stability subnets; guard against VC not having checked an epoch ahead, fixing a crash; clarify unsigned comparison

* revert unsub fix
 2020-12-22 10:05:36 +01:00
Jacek Sieka
6c8f630170
Revert "have each validator randomly pick a stability subnet, per spec (#2194)" 
This reverts commit 048a67d525cf6185d747353919897b95c0adf3de.

Fails with:
```
Error: unhandled exception: /data/beacon-node-builds/devel-large/repo/beacon_chain/nimbus_beacon_node.nim(442, 12) `node.attestationSubnets.stabilitySubnets.len == 0`  [AssertionError]
```
 2020-12-18 23:04:31 +01:00
Jacek Sieka
0f8a3a5ae8
checkpoint database at end of each slot (#2195) 
* checkpoint database at end of each slot

To avoid spending time on synchronizing with the file system while doing
processing, the manual checkpointing mode turns off fsync during
processing and instead checkpoints the database when the slot has ended.

From an sqlite perspecitve, in WAL mode this guaranees database
consistency but may lead to data loss which is fine - anything missing
from the beacon chain database can be recovered on the next startup.

* log sync status and delay in slot start message

* bump
 2020-12-18 22:01:24 +01:00
Jacek Sieka
452042b17b
msgid formatting, libp2p bump (#2198) 
* msgid formatting, libp2p bump

* fix libp2p commit
 2020-12-18 18:45:27 +00:00
tersec
048a67d525
have each validator randomly pick a stability subnet, per spec (#2194) 2020-12-18 15:46:07 +01:00
Kim De Mey
8cc7effe52
Fix ENR attnets update to only hold persistent subnets (#2193) 
* Fix ENR attnets update to only hold persistent subnets

* Use only stability subnet in metadata and enr
 2020-12-18 09:50:29 +01:00
Zahary Karadjov
7d95e86c50
Merge branch 'stable' into devel 2020-12-16 22:22:21 +02:00
Zahary Karadjov
f06e6d46a2
v1.0.4 2020-12-16 22:20:45 +02:00
Jacek Sieka
5d8cdb88c6
update validator metrics on startup 2020-12-16 20:44:48 +02:00
Jacek Sieka
de779c7812 update validator metrics on startup 2020-12-16 19:42:19 +02:00
Jacek Sieka
fea2b712f0
cache not needed to validate exits (#2188) 2020-12-16 15:36:02 +01:00
Jacek Sieka
7d5edb4353
use new stew helpers for assignment (#2172) 
* bump libp2p (reduces libp2p gossip memory usage to ~1/3)
* use "generic" assign version
 2020-12-16 09:37:22 +01:00
Zahary Karadjov
c56677867b
Remove some accidentally left debugging changes 2020-12-16 01:09:19 +02:00
Zahary Karadjov
8ebf9c30b0
More complete reset of the web3 provider on each failure; Fix #2184 2020-12-16 00:21:11 +02:00
Jacek Sieka
d59fc8ff5b
readd os import, sometimes used 2020-12-15 16:34:56 +01:00
tersec
91786686d5
don't repeat already-included attestations (#2061) 
Don't repeat already-included attestations

Also removes the superfluous (and badly scaling) attestation-cache-eviction
 2020-12-15 17:16:10 +02:00
Jacek Sieka
4e191a06ac use stew/leb128 
* 3 varint parsers -> 1
* 1/3 memory usage / allocations when writing stuff to libp2p
 2020-12-15 17:06:15 +02:00