status-im/nimbus-eth2
2
0
Fork 0
mirror of https://github.com/status-im/nimbus-eth2.git synced 2025-01-11 14:54:12 +00:00
Code Issues Projects Releases Wiki Activity
4,018 Commits 108 Branches 116 Tags 245 MiB
Commit Graph

190 Commits

Author SHA1 Message Date
tersec
68ca600d89
update 20 more spec refs to v1.1.0-alpha.8 (#2690) 2021-07-01 12:13:59 +00:00
tersec
1c3314f08b
update to Altair as of v1.1.0-alpha.7 (#2649) 
* update to Altair as of v1.1.0-alpha.7

* introduce Altair types into attestation pool

* avoid allocating/copying pubkeys excessively in get_next_sync_committee()
 2021-06-14 17:42:46 +00:00
tersec
8ebd496fbe
Altair transition tests (#2624) 
* Working Altair transition tests

* with fixed upstream test vectors, remove state root workaround

* switch upgrade_to_altair() to returning a reference

* remove test_state_transition

* fix invalid fork state/block combinations error messages

* avoid memory copies by reintroducing state_transition_slots(var SomeHashedBeaconState)
 2021-06-04 10:38:00 +00:00
tersec
28a5bca71a
split state_transition() into slots/block parts and use only block where appropriate (#2630) 2021-06-03 11:42:25 +02:00
Jacek Sieka
abe0d7b4ae singe validator key cache 
Instead of keeping a validator key list per EpochRef, this PR introduces
a single shared validator key list in ChainDAG, and cleans up some other
ChainDAG and key-related issues.

The PR does not introduce the validator key list in the state transition
- this is because we batch-check all signatures before entering the spec
code, thus the spec code never hits the cache.

A future refactor should _probably_ remove the threadvar altogether.

There's a few other small fixes in here that make the flow easier to
read:

* fix `var ChainDAGRef` -> `ChainDAGRef`
* fix `var QuarantineRef` -> `QuarantineRef`
* consistent `dag` variable name
* avoid using threadvar pubkey cache in most cases
* better error messages in batch signature checking
 2021-06-01 20:43:44 +03:00
tersec
c06ffc7804
proposed structure for altair (#2323) 
* proposed structure for hf1

* refactor datatypes.nim into datatypes/{base, phase0, hf1}.nim

* hf1 is Altair

* some syncing with alpha 2

* adjust epoch processing to disambiguate access to RewardFlags

* relocate StateData to stay consistent with meaning phase 0 StateData

* passes v1.1.0 alpha 5 SSZ consensus object tests

* Altair block header test fixtures work

* fix slash_validator() so that Altair attester slashings, proposer slashings, and voluntary exit textures work

* deposit operation Altair test fixtures work

* slot sanity and all but a couple epoch transition tests switched to Altair

* attestation Altair test fixtures work

* Altair block sanity test fixtures work

* add working altair sync committee tests

* improve workarounds for sum-types-across-modules Nim bug; incorporate SignedBeaconBlock root reconstuction to SSZ byte reader
 2021-05-28 15:25:58 +00:00
Jacek Sieka
427c0f307c
avoid extraneous hash root calculation (#2537) 
When applying a block, we'll currently compute a state root for the
state after slot processing but before block processing - this is
unnecessary when a block is being applied because the intermediate state
root is never observed.
 2021-05-05 08:54:21 +02:00
Jacek Sieka
7165e0ac31
Reset cached indices when resetting cache on SSZ read (#2480) 
* Reset cached indices when resetting cache on SSZ read

When deserializing into an existing structure, the cache should be
cleared - goes for json also. Also improve error messages.
 2021-04-08 13:11:04 +03:00
tersec
97f7284e51
bump spec refs from v1.0.0 to v1.0.1 and update copyright years (#2357) 2021-02-25 13:37:22 +00:00
tersec
1bdbf099cc
use IntSet rather than HashSet[ValidatorIndex] (#2267) 
* use IntSet rather than HashSet[ValidatorIndex]

* add bounds check before uint64 -> int conversion

* use intsets in block transitions

* remove superfluous Nim issue explanation/reference
 2021-01-26 12:52:00 +01:00
Mamy Ratsimbazafy
70a03658e3
Block validation flow v2 + Batch (serial) sig verification (#2250) 
* bump nim-blscurve

* Outline the block validation flow

* introduce the SigVerified types, pass the tests

* Split clearance/quarantine to prepare for batch crypto verif

* Add a batch signature collector

* Make clearance use SigVerified block and split verification between crypto and state transition

* Always use signedBeaconBlock for the onBlockAdded callback

* RANDAO signing_root is the epoch instead of the full block

* Support skipping BLS for testing

* Fix compilation of the validator client

* Try to fix strange errors MacOS and Jenkins (Clang, unknown type name br_hmac_drbg_context in stdlib_assertions.nim.c)

* address https://github.com/status-im/nimbus-eth2/pull/2250#discussion_r561819858

* address https://github.com/status-im/nimbus-eth2/pull/2250#discussion_r561828025

* onBlockAdded callback should use TrustedSignedBeaconBlock https://github.com/status-im/nimbus-eth2/pull/2250#discussion_r561837261

* address https://github.com/status-im/nimbus-eth2/pull/2250#discussion_r561828946

* Use the application RNG: https://github.com/status-im/nimbus-eth2/pull/2250#discussion_r561815336

* Improve codegen of conversion zero-cost)

* Quick fixes with loadWithCache after #2259 (TODO: graceful error since pubkey validations is now done first in signatures_batch)

* Graceful handle rogue pubkeys and signatures now that those are lazy-loaded
 2021-01-25 20:45:48 +02:00
Jacek Sieka
5713a3ce4c
performance fixes (#2259) 
* performance fixes

* don't mark tree cache as dirty on read-only List accesses
* store only blob in memory for keys and signatures, parse blob lazily
* compare public keys by blob instead of parsing / converting to raw
* compare Eth2Digest using non-constant-time comparison
* avoid some unnecessary validator copying

This branch will in particular speed up deposit processing which has
been slowing down block replay.

Pre (mainnet, 1600 blocks):

```
All time are ms
     Average,       StdDev,          Min,          Max,      Samples,         Test
Validation is turned off meaning that no BLS operations are performed
    3450.269,        0.000,     3450.269,     3450.269,            1, Initialize DB
       0.417,        0.822,        0.036,       21.098,         1400, Load block from database
      16.521,        0.000,       16.521,       16.521,            1, Load state from database
      27.906,       50.846,        8.104,     1507.633,         1350, Apply block
      52.617,       37.029,       20.640,      135.938,           50, Apply epoch block
```

Post:

```
    3502.715,        0.000,     3502.715,     3502.715,            1, Initialize DB
       0.080,        0.560,        0.035,       21.015,         1400, Load block from database
      17.595,        0.000,       17.595,       17.595,            1, Load state from database
      15.706,       11.028,        8.300,      107.537,         1350, Apply block
      33.217,       12.622,       17.331,       60.580,           50, Apply epoch block
```

* more perf fixes

* load EpochRef cache into StateCache more aggressively
* point out security concern with public key cache
* reuse proposer index from state when processing block
* avoid genericAssign in a few more places
* don't parse key when signature is unparseable
* fix `==` overload for Eth2Digest
* preallocate validator list when getting active validators
* speed up proposer index calculation a little bit
* reuse cache when replaying blocks in ncli_db
* avoid a few more copying loops

```
     Average,       StdDev,          Min,          Max,      Samples,         Test
Validation is turned off meaning that no BLS operations are performed
    3279.158,        0.000,     3279.158,     3279.158,            1, Initialize DB
       0.072,        0.357,        0.035,       13.400,         1400, Load block from database
      17.295,        0.000,       17.295,       17.295,            1, Load state from database
       5.918,        9.896,        0.198,       98.028,         1350, Apply block
      15.888,       10.951,        7.902,       39.535,           50, Apply epoch block
       0.000,        0.000,        0.000,        0.000,            0, Database block store
```

* clear full balance cache before processing rewards and penalties

```
All time are ms
     Average,       StdDev,          Min,          Max,      Samples,         Test
Validation is turned off meaning that no BLS operations are performed
    3947.901,        0.000,     3947.901,     3947.901,            1, Initialize DB
       0.124,        0.506,        0.026,      202.370,       363345, Load block from database
      97.614,        0.000,       97.614,       97.614,            1, Load state from database
       0.186,        0.188,        0.012,       99.561,       357262, Advance slot, non-epoch
      14.161,        5.966,        1.099,      395.511,        11524, Advance slot, epoch
       1.372,        4.170,        0.017,      276.401,       363345, Apply block, no slot processing
       0.000,        0.000,        0.000,        0.000,            0, Database block store
```
 2021-01-25 13:04:18 +01:00
Jacek Sieka
fea2b712f0
cache not needed to validate exits (#2188) 2020-12-16 15:36:02 +01:00
Zahary Karadjov
f4a3a36731
More specific error message when block validation fails 2020-12-03 11:51:42 +02:00
tersec
d5e3712609
log errors in more detail when block production fails (#2135) 
* log errors in more detail when block production fails

* pass through block header error

* make everything cstring
 2020-12-02 13:23:10 +01:00
Jacek Sieka
e7f2735271
fix broken metrics during replay (#2090) 
* move metrics out of state transition
* add validator count metric
* remove expensive beacon_current_validators, beacon_previous_validators
metrics (they should be reimplemented with cache), add cheap
beacon_active_validators to approximate
* remove unused validator count metrics
* tidy imports/defects
 2020-11-27 23:16:13 +01:00
tersec
271df8b604
bump 1.0.0rc-0 spec refs to 1.0.0 (#1974) 2020-11-09 14:18:55 +00:00
Jacek Sieka
95f5f76180
Datatype cleanup (#1953) 
* clear up spec todo

* test fix

* remove unnecessary toSszType

* type

* one more
 2020-11-04 21:52:47 +00:00
Jacek Sieka
ca63d48b82
tone down validator exit logging (#1939) 
Validators exiting is normal, no need to scream about it

* avoid reallocating seq on big exit queue
* avoid fetching state cache when updating head (it's rarely needed)
* remove incorrectly implemented live validator counts (avoids memory
allocs)
 2020-11-02 18:34:23 +01:00
tersec
1994ffe5a0
update 130+ spec references from v0.12.3 to v1.0.0-rc1 (#1854) 2020-10-12 08:59:24 +00:00
Mamy Ratsimbazafy
0280d6c73e
Revisiting log levels (#1788) 
* Update log level - https://github.com/status-im/nim-beacon-chain/issues/1779 https://github.com/status-im/nim-beacon-chain/issues/1785

* Address review comments

* Document the logging strategy [skip ci]
 2020-10-01 20:56:42 +02:00
tersec
f96ad87d28
switch another 50+ spec refs from v0.12.2 to v0.12.3 (#1749) 2020-09-25 11:52:50 +00:00
tersec
6cf7e837ce
refactor exit pool and gossipsub-subscribe to exit pool message types (#1733) 
* refactor exit pool and gossipsub-subscribe to all exit pool message types

* remove unused loop counter
 2020-09-24 19:05:49 +02:00
tersec
9f21bbd666
[WIP] skeleton of attester slashing pool & validators (#1639) 
* skeleton of attester slashing pool & validators

* add skeleton for proposer slashings and voluntary exits; rename pool to more inclusive exit pool to stay consistent with all three; ensure is initialized by beacon_node so is safe to merge, even if it doesn't do much yet
 2020-09-14 16:26:31 +02:00
Jacek Sieka
124ec4b737
perform slashing check before mutation (#1594) 
avoid copy of validator as well
 2020-09-01 09:49:55 +02:00
tersec
83667dddfe
harden beacon_pending_deposits metrics calculation against overflow (#1566) 
* harden beacon_pending_deposits metrics calculation

* ...

* move beacon_pending_deposits and beacon_processed_deposits_total out of specs and into chain DAG
 2020-08-26 17:25:39 +02:00
Dustin Brody
99e330d014 fix underflow in deposit procesing (#1542) 2020-08-21 19:00:36 +03:00
tersec
611c5097cc
use cache in process_voluntary_exit() (#1507) 2020-08-14 12:42:59 +00:00
Jacek Sieka
a1bd44f4b0
small spec cleanups (#1501) 
* clean up logging a bit
* return error on indexed attestation check
 2020-08-13 13:47:06 +00:00
tersec
dad3dd5809
update 111 v0.12.1 beacon chain spec refs to v0.12.2 (#1380) 2020-07-27 18:49:46 +02:00
tersec
90708a8287
Prefer converting int` to uint64 and switch foo.len.uint64 to .len64 (#1375) 
* avoid converting from uint64 to int, and where most feasible, int type conversion at all

* .len.uint64 -> .len64

* fix 32-bit compilation

* try keeping state_sim loop variable/bounds as int for 32-bit Azure

* len64 -> lenu64
 2020-07-26 20:55:48 +02:00
tersec
26e893ffc2
restore EpochRef and flush statecaches on epoch transitions (#1312) 
* restore EpochRef and flush statecaches on epoch transitions

* more targeted cache invalidation

* remove get_empty_per_epoch_cache(); implement simpler but still faster get_beacon_proposer_index()/compute_proposer_index() approach; add some abstraction layer for accessing the shuffled validator indices cache

* reduce integer type conversions

* remove most of rest of integer type conversion in compute_proposer_index()
 2020-07-15 12:44:18 +02:00
tersec
4aefd8b021
fix over/underflow in proposerslashing processing (#1325) 2020-07-15 10:38:45 +00:00
zah
0be77f9cbc
Restore compilation with custom presets (#1309) 
This also assigns precise types to the constants in the minimal
and mainnet presets in order to reduce the chance of compilation
errors when custom presets are used (previously, only the custom
presets have precisely assigned types for the constants).
 2020-07-13 16:44:58 +02:00
Zahary Karadjov
3ec6a02b12
Merge devel and resolve conflicts 2020-07-10 02:02:40 +03:00
Dustin Brody
4140b3b9d9 update 29 spec refs to v0.12.1 2020-07-08 20:49:25 +00:00
Zahary Karadjov
318b225ccd
Merge devel and resolve the conflicts 2020-07-08 15:36:03 +03:00
Zahary Karadjov
c4af4e2f35
Working test suite with run-time presets 2020-07-08 02:02:14 +03:00
tersec
a92276d510
adopt Result[void, string] in place of some bool return signatures (#1275) 
* adopt Result[void, string] in place of some bool return signatures

* string -> cstring to reduce memory allocations; ensure all err() strings are constants, with contextual information from higher-level callers

* logScope usage fixes

* homogenize err() reporting convention

* invalid signature in deposit isn't an error
 2020-07-03 17:03:14 +00:00
tersec
b4db2ad693
remove v0.11.3 support; add block_sim to CI (#1253) 
* remove v0.11.3 support; add block_sim to CI

* rm stray PERSISTENT_COMMITTEE_PERIOD

* remove TopicFilter.InteropAttestations

* bump two comment-spec-refs to v0.12.1
 2020-06-29 18:08:58 +00:00
Jacek Sieka
1301600341
Trusted blocks (#1227) 
* cleanups

* fix ncli state root check flag
* add block dump to ncli_db
* limit ncli_db benchmark length
* tone down finalization logs

* introduce trusted blocks

We only store blocks whose signature we've verified in the database - as
such, there's no need to check it again, and most importantly, no need
to deserialize the signature when loading from database.

50x startup time improvement, 200x block load time improvement.

* fix rewinding when deposits have invalid signature
* speed up ancestor iteration by avoiding copy
* avoid deserializing signatures for trusted data
* load blocks lazily when rewinding (less memory used)

* chronicles workarounds

* document trustedbeaconblock
 2020-06-25 12:23:10 +02:00
Dustin Brody
db870fead4 update 20 beacon chain protocol spec refs 2020-06-18 08:39:28 +00:00
tersec
ee9f4a2e3f
remove skipMerkleValidation and skipBlockParentRootValidation (#1197) 2020-06-18 07:56:47 +02:00
Dustin Brody
ffca27b45f update 24 v0.11.x spec refs to v0.12.1 2020-06-17 12:11:03 +00:00
Dustin Brody
ec85a8bdbf update a dozen unchanged spec refs to v0.12.1 2020-06-17 09:25:54 +00:00
Jacek Sieka
5c25d23ef1 eth2hash -> eth2digest 
hash in nim is the insecure hash-map helper - might as well use `digest`
consistently to mark the difference
 2020-06-16 13:30:21 +00:00
Jacek Sieka
89e4819ce9
collect signature production and verificaiton in one place (#1179) 
* collect signature production and verificaiton in one place

Signatures are made over data and domain - here we collect all such
activities in one place.

Also:
* security: fix cast-before-range-check
* log block/attestation verification consistently
* run block verification based on `getProposer` in its own history
* clean up some unused stuff

* import

* missing raises
 2020-06-16 07:45:04 +02:00
Dustin Brody
8a2c796e83 switch 12 unchanged spec references from 0.11.x to 0.12.1; line-wrap and remove pointless-to-counterproductive "return" 2020-06-15 10:10:31 +00:00
Dustin Brody
1d1e90adf4 Update 36 spec refs from 0.11.x to 0.12.1 2020-06-12 19:41:03 +00:00
Dustin Brody
8f7541959a allow compile-time switching between v0.11.3 and v0.12.1 2020-06-09 18:39:05 +00:00