status-im/nimbus-eth2
2
0
Fork 0
mirror of https://github.com/status-im/nimbus-eth2.git synced 2025-01-10 14:26:26 +00:00
Code Issues Projects Releases Wiki Activity
3,002 Commits 108 Branches 116 Tags 244 MiB
Commit Graph

69 Commits

Author SHA1 Message Date
Dustin Brody
707fcd99cb remove unused beacon chain spec and test code 2021-02-02 14:56:38 +02:00
Dustin Brody
490e4c7666 don't used sorted() in get_indexed_attestation() 2021-01-30 00:23:02 +02:00
tersec
1bdbf099cc
use IntSet rather than HashSet[ValidatorIndex] (#2267) 
* use IntSet rather than HashSet[ValidatorIndex]

* add bounds check before uint64 -> int conversion

* use intsets in block transitions

* remove superfluous Nim issue explanation/reference
 2021-01-26 12:52:00 +01:00
Jacek Sieka
5713a3ce4c
performance fixes (#2259) 
* performance fixes

* don't mark tree cache as dirty on read-only List accesses
* store only blob in memory for keys and signatures, parse blob lazily
* compare public keys by blob instead of parsing / converting to raw
* compare Eth2Digest using non-constant-time comparison
* avoid some unnecessary validator copying

This branch will in particular speed up deposit processing which has
been slowing down block replay.

Pre (mainnet, 1600 blocks):

```
All time are ms
     Average,       StdDev,          Min,          Max,      Samples,         Test
Validation is turned off meaning that no BLS operations are performed
    3450.269,        0.000,     3450.269,     3450.269,            1, Initialize DB
       0.417,        0.822,        0.036,       21.098,         1400, Load block from database
      16.521,        0.000,       16.521,       16.521,            1, Load state from database
      27.906,       50.846,        8.104,     1507.633,         1350, Apply block
      52.617,       37.029,       20.640,      135.938,           50, Apply epoch block
```

Post:

```
    3502.715,        0.000,     3502.715,     3502.715,            1, Initialize DB
       0.080,        0.560,        0.035,       21.015,         1400, Load block from database
      17.595,        0.000,       17.595,       17.595,            1, Load state from database
      15.706,       11.028,        8.300,      107.537,         1350, Apply block
      33.217,       12.622,       17.331,       60.580,           50, Apply epoch block
```

* more perf fixes

* load EpochRef cache into StateCache more aggressively
* point out security concern with public key cache
* reuse proposer index from state when processing block
* avoid genericAssign in a few more places
* don't parse key when signature is unparseable
* fix `==` overload for Eth2Digest
* preallocate validator list when getting active validators
* speed up proposer index calculation a little bit
* reuse cache when replaying blocks in ncli_db
* avoid a few more copying loops

```
     Average,       StdDev,          Min,          Max,      Samples,         Test
Validation is turned off meaning that no BLS operations are performed
    3279.158,        0.000,     3279.158,     3279.158,            1, Initialize DB
       0.072,        0.357,        0.035,       13.400,         1400, Load block from database
      17.295,        0.000,       17.295,       17.295,            1, Load state from database
       5.918,        9.896,        0.198,       98.028,         1350, Apply block
      15.888,       10.951,        7.902,       39.535,           50, Apply epoch block
       0.000,        0.000,        0.000,        0.000,            0, Database block store
```

* clear full balance cache before processing rewards and penalties

```
All time are ms
     Average,       StdDev,          Min,          Max,      Samples,         Test
Validation is turned off meaning that no BLS operations are performed
    3947.901,        0.000,     3947.901,     3947.901,            1, Initialize DB
       0.124,        0.506,        0.026,      202.370,       363345, Load block from database
      97.614,        0.000,       97.614,       97.614,            1, Load state from database
       0.186,        0.188,        0.012,       99.561,       357262, Advance slot, non-epoch
      14.161,        5.966,        1.099,      395.511,        11524, Advance slot, epoch
       1.372,        4.170,        0.017,      276.401,       363345, Apply block, no slot processing
       0.000,        0.000,        0.000,        0.000,            0, Database block store
```
 2021-01-25 13:04:18 +01:00
tersec
7f5a26002d
remove some skipBlsValidation usage/handling in tests (#2258) 2021-01-22 14:29:04 +01:00
tersec
1d7fb2ed0c
remove {.inline.} pragmas (#2033) 
* remove {.inline.} pragmas

* re-add inline on bitseqs functions and tweak inlining threshold

* remove macOS/LLVM inlining setting; revert non-init/module-local/tests inline pragma removal
 2020-11-20 11:00:22 +01:00
tersec
21c4ce8fd4
remove superfluous TODOs/not-really-TODOs, type conversion, imports (#2025) 2020-11-16 17:10:51 +01:00
Zahary Karadjov
b022dc4d1f Use O(n) algorithm in initialize_beacon_state_from_eth1; Avoid unnecessary merkle proofs generation 2020-11-15 21:40:40 +02:00
tersec
b9e122a60c
address some TODOs (#2005) 2020-11-12 19:24:07 +00:00
Zahary Karadjov
14b2d4324d openarray -> openArray 2020-11-03 23:23:10 +02:00
tersec
3d5f24f14c
stop discarding future epochs; remove a StateCache() construction (#1610) 
* stop discarding non-existent future epochs during epoch state transitions; remove a pointless StateCache() construction in advance_slots()

* update nbench to pass StateCache to process_slots()
 2020-09-07 15:04:33 +00:00
Mamy Ratsimbazafy
454b9d0724
Bump nim-blscurve (#1491) 
* Bump BLSCurve

* Use unified aggregation API

* use new blscurve with unified aggregate API

* bump

* fix toRaw

* replace state_sim combine with AggregateSignature

* Fix 32-bit

* Fix 32-bit for real and test deactivating ccache for fno-tree-lopp-vectorize flag

* change compilation switches to narrow down Linux issue

* Use -fno-tree-vectorize to disable both tree-loop-vectorize and tree-slp-vectorize

* blscurve now disables both Loop and SLP vectorization

* Add tests for the miracl/milagro fallback

* Travis has max log size of 4MB

* Test with Miracl in the finalization test

* fix state_sim log level

* Coment out the slow fallback tests
 2020-08-15 19:33:58 +02:00
Jacek Sieka
cf8cd8321b
Revert "Lazy crypto [alt design #1369] (#1371)" (#1435) 
This reverts commit 023f7f4518fb8dd9cab2af5faba76b6975893dea.
 2020-08-04 17:15:13 +00:00
Mamy Ratsimbazafy
023f7f4518
Lazy crypto [alt design #1369] (#1371) 
* Lazy loading of crypto objects

* Try to fix incorrect field access by hiding fields but no luck. SSZ/Chronicles/macro bug?

* Fix incorrect blsValue access. was "aggregate" not "chronicles"

* Fix tests that rely on the internal BLSValue representation
 2020-07-29 18:13:05 +00:00
tersec
90708a8287
Prefer converting int` to uint64 and switch foo.len.uint64 to .len64 (#1375) 
* avoid converting from uint64 to int, and where most feasible, int type conversion at all

* .len.uint64 -> .len64

* fix 32-bit compilation

* try keeping state_sim loop variable/bounds as int for 32-bit Azure

* len64 -> lenu64
 2020-07-26 20:55:48 +02:00
Jacek Sieka
8b01284b0e
cache block hash (#1329) 
hash_tree_root was turning up when running beacon_node, turns out to be
repeated hash_tree_root invocations - this pr brings them back down to
normal.

this PR caches the root of a block in the SignedBeaconBlock object -
this has the potential downside that even invalid blocks will be hashed
(as part of deserialization) - later, one could imagine delaying this
until checks have passed

there's also some cleanup of the `cat=` logs which were applied randomly
and haphazardly, and to a large degree are duplicated by other
information in the log statements - in particular, topics fulfill the
same role
 2020-07-16 15:16:51 +02:00
tersec
26e893ffc2
restore EpochRef and flush statecaches on epoch transitions (#1312) 
* restore EpochRef and flush statecaches on epoch transitions

* more targeted cache invalidation

* remove get_empty_per_epoch_cache(); implement simpler but still faster get_beacon_proposer_index()/compute_proposer_index() approach; add some abstraction layer for accessing the shuffled validator indices cache

* reduce integer type conversions

* remove most of rest of integer type conversion in compute_proposer_index()
 2020-07-15 12:44:18 +02:00
Zahary Karadjov
3ec6a02b12
Merge devel and resolve conflicts 2020-07-10 02:02:40 +03:00
tersec
61b0b5af17
update most remaining non-fork-choice spec refs, updating code where necessary (#1292) 
* update most of the remaining non-fork-choice spec refs, updating code where necessary

* revert presumably harmless compute_signing_root() change, but this way, keep things really unchanged outside inspector
 2020-07-09 11:43:27 +00:00
Zahary Karadjov
318b225ccd
Merge devel and resolve the conflicts 2020-07-08 15:36:03 +03:00
tersec
b7d1967449
test for correctly skipping, without failure, deposits with invalid signatures (#1288) 2020-07-08 06:55:23 +02:00
Zahary Karadjov
c4af4e2f35
Working test suite with run-time presets 2020-07-08 02:02:14 +03:00
Jacek Sieka
6fe0a623f5
Crypto rng (#1284) 
* use bearssl rng throughout

* bump

* bump

* move keygen out of crypto
 2020-07-07 17:51:02 +02:00
tersec
c64737e7f2
implement aggregated attestation receiving/validating (#1272) 
* implement aggregated attestation receiving/validating

* document the conditions without explicit implementations in isValidAggregatedAttestation()
 2020-07-02 16:15:27 +00:00
tersec
807b920c19
state_transition implements the spec fairly directly (#1220) 2020-06-23 13:54:24 +00:00
Jacek Sieka
89e4819ce9
collect signature production and verificaiton in one place (#1179) 
* collect signature production and verificaiton in one place

Signatures are made over data and domain - here we collect all such
activities in one place.

Also:
* security: fix cast-before-range-check
* log block/attestation verification consistently
* run block verification based on `getProposer` in its own history
* clean up some unused stuff

* import

* missing raises
 2020-06-16 07:45:04 +02:00
tersec
c8f24ae3b8
Remove three skipMerkleValidation usages (#1164) 
* remove three skipMerkleValidation usages

* remove a couple obsolete comments/TODOs
 2020-06-12 18:03:46 +02:00
Dustin Brody
f4d38611ef check that process_slots() succeeds 2020-05-19 17:16:55 +00:00
Dustin Brody
6c941b1ff7 spec v0.11.3 update 2020-05-19 17:16:55 +00:00
Zahary Karadjov
a99977b772 Progress towards #991; Enable the distinct ssz.List type 2020-05-19 15:53:03 +03:00
Jacek Sieka
fb2e0ddbec
sync fixes (#1005) 
* sync fixes

* fix Status message finalized info
* work around sync starting before initial status exchange
* don't fail block on deposit signature check failure (fixes #989)
* print ForkDigest and Version nicely
* dump incoming blocks
* fix crash when libp2p peer connection is closed
* update chunk size to 16 to work around missing blocks when syncing

* bump libp2p

* bump libp2p

* better deposit skip message
 2020-05-11 18:08:52 +00:00
tersec
3a56ddc5c4
begin removing plain BeaconState versions of state transition functions (#951) 
* remove near-duplicate code paths: process_slot(), process_slots(), and state_transition() for BeaconState are now wrappers around the HashedBeaconState versions

* convert tests/test_state_transition.nim to use HashedBeaconState

* convert mocking infrastructure and spec_block/epoch_processing tests to use HashedBeaconBlock, and remove thus unused process_slot*(state: var BeaconState)
 2020-04-30 08:44:19 +02:00
Jacek Sieka
03a147ab8d
avoid state copy in state transition (#930) 
In BlockPool, we keep the head state around, so it's trivial to restore
the temporary state there and keep going as if nothing happened.

This solves 3 problems:
* stack space - the state copy on mainnet is huge
* GC scanning - using stack space for state slows down the GC
significantly
* reckless copying - the copy itself takes a long time

In state_sim, we'll do the same and allocate on heap - this helps a
little with GC - without it, the collection of the temporary strings
created with `toHex` while printing the json dominates the trace.
 2020-04-26 21:13:33 +02:00
Zahary Karadjov
4e9fa51ae9 Introduce BeaconNodeRef and use it in all the right places 2020-04-26 13:04:53 +03:00
Dustin Brody
d559f4ee89 create distinct CommitteeIndex type 2020-04-15 09:59:49 +00:00
Dustin Brody
f49dbf68e6 refactor some Merkle proof generation code outside tests; remove unused variables/functions 2020-04-15 09:59:49 +00:00
Jacek Sieka
afa08c8e3c
crypto: cleanup (#882) 
* crypto: cleanup

* fix several Defect-on-user-input
* make crypto interface more similar to secp
* use `crypto.nim` in all of nbc

* digest: raises

* fix

* vendor
 2020-04-11 10:51:07 +02:00
tersec
ccbbce79a9
Mostly remove skipMerkleValidation by fixing Merkle proof construction/usage (#879) 
* refactor and fix merkle proof construction in test suite and thereby remove most remaining skipMerkleValidation flags, now unnecessary

* a few non-semantic comment update/removals
 2020-04-10 15:59:17 +02:00
Jacek Sieka
0f47c76b50
reenable block bls verification (#875) 
* finalizing state_transition

* cleanup

Co-authored-by: Dustin Brody <tersec@users.noreply.github.com>
 2020-04-09 09:41:02 +00:00
Dustin Brody
366da6df36 fix and refactor merkle_minimal sanity checks to run correctly and under CI 2020-04-08 15:40:02 +00:00
tersec
f5f939bd31
0.11.1 beacon chain spec update (#836) 
* initial 0.11.1 spec commit; no test regressions and finalizes in eth2_network_simulation

* with BLS 0.10/0.11 available, stop skipping attester slashing, proposer slashing, and voluntary exist operations fixture tests

* switch param orders to group state.{fork, genesis_validators_root}; bump spec/datatypes spec version for network purposes

* mark attestation construction and broadcast and some minimal/mainnet constants as 0.11.1-compatible; remove phase 1 sharding constants from minimal which don't exist in that preset
 2020-03-30 11:31:44 +00:00
Jacek Sieka
689bcf71c4 clean up block creation 
* consistently use state at new block slot to produce block
* factor out signature funcs
* fix missing block application test
 2020-03-22 11:15:07 +00:00
Dustin Brody
6068dd960b initial 0.11.0 spec version update 2020-03-17 12:06:56 +02:00
Mamy André-Ratsimbazafy
28dc8a6a29 Fix mock signatures/validator keys and stack smashing 2020-03-13 12:20:41 +00:00
Mamy André-Ratsimbazafy
bfd4df912e Update to BLS v0.10.1 2020-03-13 12:20:41 +00:00
Jacek Sieka
9a3db7a81f
[WIP] Fake bls at runtime (#735) 
* Initial implementation of runtime bls skipping.

Add libnfuzz skipBLSValidation handling, check that it propagates.

* Rename skipBLSValidation -> skipBlsValidation, start skipStateRootValidation
* Replace skipValidation flags with more granular flags.

Also added skipBlockParentRootValidation flag
Mainly replaced with skipBlsValidation but also StateRoot or
BlockParentRootValidation flags where appropriate.

* Adjust interop test to pass when skipping merkle validation.
* Stop skipping validation for mainchain_monitor.
* Remove comment.
* Also skipMerkleValidation for test_beacon_chain_db.
 2020-03-05 13:52:10 +01:00
Joao Gabriel Carvalho
91f87b55b4
SignedBeaconBlock in state_transition (#773) 
* using SignedBeaconBlock in state_transition
 2020-02-29 16:15:44 +01:00
Dustin Brody
1ffc2df23d add a couple new deposit tests; fix the false-positive BLS verifications while keeping all but two tests working, despite mismatched 0.9/0.10 BLS standards; better-factor the skipping of BLS validation and Merkle tree validation 2020-02-04 18:39:38 +00:00
Dustin Brody
417f96213d fix genesis interop by fixing deposit signature handling; add TRACE-level compilation to CI 2019-12-20 15:26:48 +00:00
Dustin Brody
c824416f56 initial 0.9.3 spec update 2019-12-17 00:06:03 +02:00