Commit Graph

723 Commits

Author SHA1 Message Date
Mamy Ratsimbazafy d47f53cd9d
Reorg (5/5) (#2377)
* Reorg things left into networking and gossip_processing

* time -> beacon_clock

* fix builds
2021-03-05 14:12:00 +01:00
tersec 451cc03d76
datatypes spec ref url updates (#2372) 2021-03-02 17:31:34 +01:00
tersec de643d9926
allow multiple hard fork datatypes to coexist (#2328)
* allow multiple hard fork datatypes to coexist

* update to 1.0.1

* merge recent datatypes.nim updates

* trigger rebuild now the out-of-disk-space machine offline
2021-03-02 10:13:39 +00:00
tersec 5653b2e13c
more spec v1.0.1 spec ref URL and copyright year updates (#2367) 2021-03-02 06:04:14 +00:00
tersec e661f7d0c7
prevent uint64 to int64-induced RangeError/RangeDefects in metrics (#2358)
* prevent uint64 to int64-induced RangeError/RangeDefects in metrics

* remove redundant min(foo, int64.high)

* adjust spacing to be consistent
2021-03-01 20:55:25 +01:00
tersec 97f7284e51
bump spec refs from v1.0.0 to v1.0.1 and update copyright years (#2357) 2021-02-25 13:37:22 +00:00
Dustin Brody f14e7babb6 update eth2 specs to version v1.0.1 2021-02-25 14:21:59 +02:00
Ștefan Talpalaru 44a1263ece fix Eth2Digest compile-time comparison 2021-02-25 14:20:26 +02:00
Ștefan Talpalaru 16abf2989b bump NimYAML 2021-02-25 14:20:26 +02:00
Dustin Brody c7093c4ab5 show next attestation slot & wait time in Slot end log 2021-02-15 22:49:20 +02:00
tersec 5cab17dc1a
database state storage benchmarking via ncli_db (#2312)
* database state storage benchmarking via ncli_db

* more cleanups from immutable validator state branch

* unexport some eth2_network constants and remove unused variables/templates

* make two PeerScore constants public
2021-02-15 17:40:00 +01:00
tersec aca3e4cd5c
per HF1, split process_final_updates() (#2319) 2021-02-14 19:31:01 +00:00
Jacek Sieka 9968944329
more callsigs! (#2302) 2021-02-09 10:23:26 +01:00
Jacek Sieka 8a09286423
tune attestation params (#2301)
* a little bit more priority to attestation processing
* better implementation for bytes_to_uint64
2021-02-08 16:13:02 +01:00
Ștefan Talpalaru 80c11546ff Windows binary release
CI: use both cores on GitHub Actions and set timeouts for the local testnet tests
2021-02-04 10:25:44 +02:00
Dustin Brody 707fcd99cb remove unused beacon chain spec and test code 2021-02-02 14:56:38 +02:00
Jacek Sieka 5d3c995d42 json-rpc fixes
* expose node signatures
* format bitseqs as hex strings
* format trusted sigs as hex strings (same as untrusted)
* reuse rpc client sigs
* include validator index in duties
* move SyncInfo to spec
2021-02-02 14:55:36 +02:00
Dustin Brody 490e4c7666 don't used sorted() in get_indexed_attestation() 2021-01-30 00:23:02 +02:00
Dustin Brody 281853eee8 rename options and internal structures to doppelgangerFoo and remove probing 2021-01-30 00:17:54 +02:00
tersec 1bdbf099cc
use IntSet rather than HashSet[ValidatorIndex] (#2267)
* use IntSet rather than HashSet[ValidatorIndex]

* add bounds check before uint64 -> int conversion

* use intsets in block transitions

* remove superfluous Nim issue explanation/reference
2021-01-26 12:52:00 +01:00
Mamy Ratsimbazafy 70a03658e3
Block validation flow v2 + Batch (serial) sig verification (#2250)
* bump nim-blscurve

* Outline the block validation flow

* introduce the SigVerified types, pass the tests

* Split clearance/quarantine to prepare for batch crypto verif

* Add a batch signature collector

* Make clearance use SigVerified block and split verification between crypto and state transition

* Always use signedBeaconBlock for the onBlockAdded callback

* RANDAO signing_root is the epoch instead of the full block

* Support skipping BLS for testing

* Fix compilation of the validator client

* Try to fix strange errors MacOS and Jenkins (Clang, unknown type name br_hmac_drbg_context in stdlib_assertions.nim.c)

* address https://github.com/status-im/nimbus-eth2/pull/2250#discussion_r561819858

* address https://github.com/status-im/nimbus-eth2/pull/2250#discussion_r561828025

* onBlockAdded callback should use TrustedSignedBeaconBlock https://github.com/status-im/nimbus-eth2/pull/2250#discussion_r561837261

* address https://github.com/status-im/nimbus-eth2/pull/2250#discussion_r561828946

* Use the application RNG: https://github.com/status-im/nimbus-eth2/pull/2250#discussion_r561815336

* Improve codegen of conversion zero-cost)

* Quick fixes with loadWithCache after #2259 (TODO: graceful error since pubkey validations is now done first in signatures_batch)

* Graceful handle rogue pubkeys and signatures now that those are lazy-loaded
2021-01-25 20:45:48 +02:00
Jacek Sieka 5ca10d3de2 avoid copying keys for trusted signatures
this removes `is_valid_indexed_attestion` from performance benchmarks
when replaying blocks
2021-01-25 19:48:02 +02:00
tersec 7d74d3bfbc
only subscribe to subnets when aggregating (#2254)
Only subscribe to subnets when aggregating
2021-01-25 19:39:56 +02:00
Jacek Sieka 5713a3ce4c
performance fixes (#2259)
* performance fixes

* don't mark tree cache as dirty on read-only List accesses
* store only blob in memory for keys and signatures, parse blob lazily
* compare public keys by blob instead of parsing / converting to raw
* compare Eth2Digest using non-constant-time comparison
* avoid some unnecessary validator copying

This branch will in particular speed up deposit processing which has
been slowing down block replay.

Pre (mainnet, 1600 blocks):

```
All time are ms
     Average,       StdDev,          Min,          Max,      Samples,         Test
Validation is turned off meaning that no BLS operations are performed
    3450.269,        0.000,     3450.269,     3450.269,            1, Initialize DB
       0.417,        0.822,        0.036,       21.098,         1400, Load block from database
      16.521,        0.000,       16.521,       16.521,            1, Load state from database
      27.906,       50.846,        8.104,     1507.633,         1350, Apply block
      52.617,       37.029,       20.640,      135.938,           50, Apply epoch block
```

Post:

```
    3502.715,        0.000,     3502.715,     3502.715,            1, Initialize DB
       0.080,        0.560,        0.035,       21.015,         1400, Load block from database
      17.595,        0.000,       17.595,       17.595,            1, Load state from database
      15.706,       11.028,        8.300,      107.537,         1350, Apply block
      33.217,       12.622,       17.331,       60.580,           50, Apply epoch block
```

* more perf fixes

* load EpochRef cache into StateCache more aggressively
* point out security concern with public key cache
* reuse proposer index from state when processing block
* avoid genericAssign in a few more places
* don't parse key when signature is unparseable
* fix `==` overload for Eth2Digest
* preallocate validator list when getting active validators
* speed up proposer index calculation a little bit
* reuse cache when replaying blocks in ncli_db
* avoid a few more copying loops

```
     Average,       StdDev,          Min,          Max,      Samples,         Test
Validation is turned off meaning that no BLS operations are performed
    3279.158,        0.000,     3279.158,     3279.158,            1, Initialize DB
       0.072,        0.357,        0.035,       13.400,         1400, Load block from database
      17.295,        0.000,       17.295,       17.295,            1, Load state from database
       5.918,        9.896,        0.198,       98.028,         1350, Apply block
      15.888,       10.951,        7.902,       39.535,           50, Apply epoch block
       0.000,        0.000,        0.000,        0.000,            0, Database block store
```

* clear full balance cache before processing rewards and penalties

```
All time are ms
     Average,       StdDev,          Min,          Max,      Samples,         Test
Validation is turned off meaning that no BLS operations are performed
    3947.901,        0.000,     3947.901,     3947.901,            1, Initialize DB
       0.124,        0.506,        0.026,      202.370,       363345, Load block from database
      97.614,        0.000,       97.614,       97.614,            1, Load state from database
       0.186,        0.188,        0.012,       99.561,       357262, Advance slot, non-epoch
      14.161,        5.966,        1.099,      395.511,        11524, Advance slot, epoch
       1.372,        4.170,        0.017,      276.401,       363345, Apply block, no slot processing
       0.000,        0.000,        0.000,        0.000,            0, Database block store
```
2021-01-25 13:04:18 +01:00
Dustin Brody a16f5afcd5 pre-emptive duplicate validator detection heuristic 2021-01-21 16:03:02 +02:00
tersec 55ecb61c3a cycle attestation subnets every slot (#2240)
Cycle attestation subnets every slot
2021-01-19 19:44:03 +02:00
tersec 921fe5a68f
initial infrastructure for state diffs (#2087)
Initial infrastructure for state diffs
2021-01-18 22:34:41 +02:00
tersec fa75c477cd
only initially subscribe to relevant attestation subnets (#2231) 2021-01-14 09:43:21 +01:00
tersec 0fad1b6b26
don't special-case zero-validator subnet cycling (#2230) 2021-01-12 17:17:43 +01:00
Zahary Karadjov 338428cbd7 Add Eth1 deposits simulation to block_sim 2021-01-04 13:22:00 +02:00
tersec 97c4f7c5c0
fix subnet calculation in RPC and insert broadcast attestations into node's pool (#2207)
* fix subnet calculation in RPC and insert broadcast attestations into node's pool

* unify codepaths to ensure only mostly-checked-to-be-valid attestations enter the pool, even from node's own broadcasts

* update attestation pool tests for new validateAttestation param
2020-12-23 13:59:04 +01:00
tersec afbaa36ef7
make subnet cycling more robust; use one stability subnet/validator; explicitly represent gossip enabled/disabled (#2201)
* make subnet cycling more robust; use one stability subnet/validator; explicitly represent gossip enabled/disabled

* fix asymmetry in _snappy being used for subscriptions but not unsubscriptions

* remove redundant comment

* minimal RPC and VC support for infoming BN of subnets

* create and verify slot signatures in RPC interface and VC

* loosen old slot check

* because Slot + uint64 works but uint64 + Slot doesn't

* document assumptions for head state use; don't clear stability subnets; guard against VC not having checked an epoch ahead, fixing a crash; clarify unsigned comparison

* revert unsub fix
2020-12-22 10:05:36 +01:00
Jacek Sieka 6c8f630170
Revert "have each validator randomly pick a stability subnet, per spec (#2194)"
This reverts commit 048a67d525.

Fails with:
```
Error: unhandled exception: /data/beacon-node-builds/devel-large/repo/beacon_chain/nimbus_beacon_node.nim(442, 12) `node.attestationSubnets.stabilitySubnets.len == 0`  [AssertionError]
```
2020-12-18 23:04:31 +01:00
tersec 048a67d525
have each validator randomly pick a stability subnet, per spec (#2194) 2020-12-18 15:46:07 +01:00
Jacek Sieka fea2b712f0
cache not needed to validate exits (#2188) 2020-12-16 15:36:02 +01:00
Jacek Sieka 7d5edb4353
use new stew helpers for assignment (#2172)
* bump libp2p (reduces libp2p gossip memory usage to ~1/3)
* use "generic" assign version
2020-12-16 09:37:22 +01:00
Eugene Kabanov 46c2740097
Documentation for Validators API. (#2147)
* Recover proper validator API call and remove incorrect one.
Add more examples to API documentation.
2020-12-07 14:51:14 +02:00
tersec 26ea76bbdf
don't lag aggregated attestations by a slot (#2150)
* don't lag aggregated attestations by a slot

* don't use aggregation topic at all

* use aggregates again, but with aggressively low ATTESTATION_PROPAGATION_SLOT_RANGE; seems to hold on to LH 1.0 nodes

* clean up scaffolding and double ATTESTATION_PROPAGATION_SLOT_RANGE to 16

* increase ATTESTATION_PROPAGATION_SLOT_RANGE to 24

* increase ATTESTATION_PROPAGATION_SLOT_RANGE to 28 and isolate in only used function due to customization; remove TRAILING_DISTANCE machinery
2020-12-07 12:56:49 +01:00
Zahary Karadjov f4a3a36731
More specific error message when block validation fails 2020-12-03 11:51:42 +02:00
cheatfate 9afafb54f5 get_v1_beacon_states_stateId_validator_balances() implementation. 2020-12-03 11:40:08 +02:00
cheatfate 5f392893c0 Change approach for get_v1_beacon_pool_attestations() to use hexadecimal strings. 2020-12-03 11:40:08 +02:00
tersec d5e3712609
log errors in more detail when block production fails (#2135)
* log errors in more detail when block production fails

* pass through block header error

* make everything cstring
2020-12-02 13:23:10 +01:00
Dustin Brody 3e4b49462b detect excessively long graffiti strings 2020-12-01 15:39:46 +02:00
Zahary Karadjov 3c0dfc2fbe Implement the 'deposits exit' command; Remove 'deposits create' 2020-11-29 23:08:07 +02:00
Jacek Sieka e7f2735271
fix broken metrics during replay (#2090)
* move metrics out of state transition
* add validator count metric
* remove expensive beacon_current_validators, beacon_previous_validators
metrics (they should be reimplemented with cache), add cheap
beacon_active_validators to approximate
* remove unused validator count metrics
* tidy imports/defects
2020-11-27 23:16:13 +01:00
Jacek Sieka d16e127daf
Peer stuff (#2084)
* Revert "Revert "Full "node" RPC calls implementation and fixes to peer lifetime states. (#2065)" (#2082)"

This reverts commit 7cc3dc8027.

* fix nil disconnectedFut crash

* fixes

don't resetPeer, it causes peer miscounts

* disconnect disconnecting peers

...when there's a race.

* avoid connection spamming

* never decrease SeenTable timeout
* only recover ENR for known peers

* seen only when really disconnected
2020-11-26 20:23:45 +01:00
Jacek Sieka 7cc3dc8027
Revert "Full "node" RPC calls implementation and fixes to peer lifetime states. (#2065)" (#2082)
This reverts commit d041287a4f.
2020-11-26 09:05:23 +01:00
Eugene Kabanov d041287a4f
Full "node" RPC calls implementation and fixes to peer lifetime states. (#2065)
* Initial commit.

* Fix log lines and compilation error.

* Add get_v1_node_peers() implementation.

* Fix peer's lifetime states.

* Use the most recent multiaddress.

* Fix assign NewPeerScore again.
Fix compilation error with last seen address.
Fix Peer upgraded log line place.

* syncing, health, peer_count, peer_id and fixes for identity.

* Fix compilation problems.

* Move object declaration to callsigs.
Fix identity addresses fields.

* Finish node RPC calls.

* Avoid leak of lifetime future.

* Bump chronos.

* Fix json generator problem.
2020-11-26 08:09:59 +01:00
Zahary Karadjov 3594fa2a22
Version 1.0.0-rc1 2020-11-25 03:13:58 +02:00
Zahary Karadjov a995cdd3d9
Bugfix: fix an incorrect balance increase during genesis state generation 2020-11-24 23:30:26 +02:00