status-im/nimbus-eth2
2
0
Fork 0
mirror of https://github.com/status-im/nimbus-eth2.git synced 2025-01-09 13:56:23 +00:00
Code Issues Projects Releases Wiki Activity
6,964 Commits 108 Branches 116 Tags 242 MiB
Commit Graph

365 Commits

Author SHA1 Message Date
Zahary Karadjov
2eacfc4685 Bump modules to take advantage of the new Json format flavors support 
Since quite a lot of additional procs were now compiled as generics, this lead to compiler bugs that had
to be worked-around:

* The `Domain` type was renamed to `Eth2Domain` to avoid compilation errors
  due to conflicts with `nativesockets.Domain`.
  Similarly, `eth2_network.KeyPair` was renamed to `NetKeyPair`.

* A new more robust version of `hexToByteArray` was added to stew
 2021-03-25 09:37:35 +02:00
tersec
97f7284e51
bump spec refs from v1.0.0 to v1.0.1 and update copyright years (#2357) 2021-02-25 13:37:22 +00:00
Jacek Sieka
8a09286423
tune attestation params (#2301) 
* a little bit more priority to attestation processing
* better implementation for bytes_to_uint64
 2021-02-08 16:13:02 +01:00
Jacek Sieka
5713a3ce4c
performance fixes (#2259) 
* performance fixes

* don't mark tree cache as dirty on read-only List accesses
* store only blob in memory for keys and signatures, parse blob lazily
* compare public keys by blob instead of parsing / converting to raw
* compare Eth2Digest using non-constant-time comparison
* avoid some unnecessary validator copying

This branch will in particular speed up deposit processing which has
been slowing down block replay.

Pre (mainnet, 1600 blocks):

```
All time are ms
     Average,       StdDev,          Min,          Max,      Samples,         Test
Validation is turned off meaning that no BLS operations are performed
    3450.269,        0.000,     3450.269,     3450.269,            1, Initialize DB
       0.417,        0.822,        0.036,       21.098,         1400, Load block from database
      16.521,        0.000,       16.521,       16.521,            1, Load state from database
      27.906,       50.846,        8.104,     1507.633,         1350, Apply block
      52.617,       37.029,       20.640,      135.938,           50, Apply epoch block
```

Post:

```
    3502.715,        0.000,     3502.715,     3502.715,            1, Initialize DB
       0.080,        0.560,        0.035,       21.015,         1400, Load block from database
      17.595,        0.000,       17.595,       17.595,            1, Load state from database
      15.706,       11.028,        8.300,      107.537,         1350, Apply block
      33.217,       12.622,       17.331,       60.580,           50, Apply epoch block
```

* more perf fixes

* load EpochRef cache into StateCache more aggressively
* point out security concern with public key cache
* reuse proposer index from state when processing block
* avoid genericAssign in a few more places
* don't parse key when signature is unparseable
* fix `==` overload for Eth2Digest
* preallocate validator list when getting active validators
* speed up proposer index calculation a little bit
* reuse cache when replaying blocks in ncli_db
* avoid a few more copying loops

```
     Average,       StdDev,          Min,          Max,      Samples,         Test
Validation is turned off meaning that no BLS operations are performed
    3279.158,        0.000,     3279.158,     3279.158,            1, Initialize DB
       0.072,        0.357,        0.035,       13.400,         1400, Load block from database
      17.295,        0.000,       17.295,       17.295,            1, Load state from database
       5.918,        9.896,        0.198,       98.028,         1350, Apply block
      15.888,       10.951,        7.902,       39.535,           50, Apply epoch block
       0.000,        0.000,        0.000,        0.000,            0, Database block store
```

* clear full balance cache before processing rewards and penalties

```
All time are ms
     Average,       StdDev,          Min,          Max,      Samples,         Test
Validation is turned off meaning that no BLS operations are performed
    3947.901,        0.000,     3947.901,     3947.901,            1, Initialize DB
       0.124,        0.506,        0.026,      202.370,       363345, Load block from database
      97.614,        0.000,       97.614,       97.614,            1, Load state from database
       0.186,        0.188,        0.012,       99.561,       357262, Advance slot, non-epoch
      14.161,        5.966,        1.099,      395.511,        11524, Advance slot, epoch
       1.372,        4.170,        0.017,      276.401,       363345, Apply block, no slot processing
       0.000,        0.000,        0.000,        0.000,            0, Database block store
```
 2021-01-25 13:04:18 +01:00
tersec
271df8b604
bump 1.0.0rc-0 spec refs to 1.0.0 (#1974) 2020-11-09 14:18:55 +00:00
Zahary Karadjov
14b2d4324d openarray -> openArray 2020-11-03 23:23:10 +02:00
tersec
1994ffe5a0
update 130+ spec references from v0.12.3 to v1.0.0-rc1 (#1854) 2020-10-12 08:59:24 +00:00
Zahary Karadjov
aed291128a Add support for starting from weak subjectivity checkpoints 
Also removes the `genesis.ssz` file stored in the data folder.
The `medalla-fast-sync` target has been adapted to use the new features.
 2020-10-07 09:32:03 +03:00
tersec
6398a43cc1
update 120+ beacon_chain and validator spec refs from v0.12.2 to v0.12.3 (#1740) 2020-09-24 19:04:10 +02:00
tersec
b3b578501a
mark comments documenting functions as a whole as such (#1613) 2020-09-08 08:54:55 +00:00
Jacek Sieka
2081c4f505
avoid unnecessary seq allocations (#1573) 
"sigh"
 2020-08-27 08:32:51 +02:00
tersec
bc6eefe31e
add --enable-logtrace argument to launch_local_testnet (#1502) 
* add --enable-logtrace argument to launch_local_testnet

* scan for all available logfiles

* remove specific filename references

* update v0.11.3 spec ref to v0.12.2
 2020-08-16 11:12:19 +02:00
tersec
dad3dd5809
update 111 v0.12.1 beacon chain spec refs to v0.12.2 (#1380) 2020-07-27 18:49:46 +02:00
Jacek Sieka
48cebc7157
spec cleanups (#1379) 
* add helper for beacon committee length (used for quickly validating
attestations)
* refactor some attestation checks to do cheap checks early
* validate attestation epoch before computing active validator set
* clean up documentation / comments
* fill state cache on demand
 2020-07-27 16:04:44 +00:00
tersec
20a2525390
v0.12.2 beacon chain protocol update (#1378) 2020-07-27 12:59:57 +02:00
tersec
90708a8287
Prefer converting int` to uint64 and switch foo.len.uint64 to .len64 (#1375) 
* avoid converting from uint64 to int, and where most feasible, int type conversion at all

* .len.uint64 -> .len64

* fix 32-bit compilation

* try keeping state_sim loop variable/bounds as int for 32-bit Azure

* len64 -> lenu64
 2020-07-26 20:55:48 +02:00
Jacek Sieka
e0a18a3105
cache beacon committee size calculation (#1363) 
* cache beacon committee size calculation

this fixes a bug in get_validator_churn_limit as well

* fix

* make committee counts consistently uint64

mixing feels like the worst of the two worlds
 2020-07-23 19:01:07 +02:00
Jacek Sieka
fb2f742972
Fork choice fixes 2 (#1356) 
* fork choice cleanup

* enable v2 pruning
* prefer `get_current_epoch`
* fix finalization check to use correct epoch

* small cleanups

* add `count_active_validators`
* remove misleading logs
* fix justified checkpoint slot calculation in rpc
 2020-07-22 23:01:44 +02:00
tersec
26e893ffc2
restore EpochRef and flush statecaches on epoch transitions (#1312) 
* restore EpochRef and flush statecaches on epoch transitions

* more targeted cache invalidation

* remove get_empty_per_epoch_cache(); implement simpler but still faster get_beacon_proposer_index()/compute_proposer_index() approach; add some abstraction layer for accessing the shuffled validator indices cache

* reduce integer type conversions

* remove most of rest of integer type conversion in compute_proposer_index()
 2020-07-15 12:44:18 +02:00
zah
0be77f9cbc
Restore compilation with custom presets (#1309) 
This also assigns precise types to the constants in the minimal
and mainnet presets in order to reduce the chance of compilation
errors when custom presets are used (previously, only the custom
presets have precisely assigned types for the constants).
 2020-07-13 16:44:58 +02:00
Zahary Karadjov
3ec6a02b12
Merge devel and resolve conflicts 2020-07-10 02:02:40 +03:00
Dustin Brody
4140b3b9d9 update 29 spec refs to v0.12.1 2020-07-08 20:49:25 +00:00
Zahary Karadjov
c4af4e2f35
Working test suite with run-time presets 2020-07-08 02:02:14 +03:00
Zahary Karadjov
441ae9b58c
Introduce a RuntimePreset object for the overridable testnet preset values 2020-07-03 22:29:23 +03:00
tersec
b4db2ad693
remove v0.11.3 support; add block_sim to CI (#1253) 
* remove v0.11.3 support; add block_sim to CI

* rm stray PERSISTENT_COMMITTEE_PERIOD

* remove TopicFilter.InteropAttestations

* bump two comment-spec-refs to v0.12.1
 2020-06-29 18:08:58 +00:00
tersec
7b4e129316
re-add minimal constant checking; organize presets by spec version (#1203) 
* re-add minimal preset constant checking; organize presets to better support multiple spec versions

* bump spec ref

* increase Azure timeout to 90 minutes to accomodate Nim compiler building
 2020-06-20 09:12:45 +02:00
tersec
dc1a565b3f
support v0.12.1 attestation topics in beacon node/inspector subscribing (#1187) 
* support v0.12.1 attestation topics in beacon node and inspector subscribing

* bump is_valid_merkle_branch() spec ref
 2020-06-18 15:10:25 +02:00
Jacek Sieka
5c25d23ef1 eth2hash -> eth2digest 
hash in nim is the insecure hash-map helper - might as well use `digest`
consistently to mark the difference
 2020-06-16 13:30:21 +00:00
Dustin Brody
1d1e90adf4 Update 36 spec refs from 0.11.x to 0.12.1 2020-06-12 19:41:03 +00:00
Dustin Brody
8f7541959a allow compile-time switching between v0.11.3 and v0.12.1 2020-06-09 18:39:05 +00:00
Dustin Brody
cc13153fc6 initial 0.12.1 beacon chain spec 2020-06-09 18:39:05 +00:00
Jacek Sieka
56ffb696be
reorder ssz (#1099) 
* reorder ssz

* split into hash_trees and ssz_serialization, roughly, for hashing and
IO
* move bitseqs into ssz (from stew)
* clean up imports

* docs, imports
 2020-06-03 15:52:02 +02:00
Viktor Kirilov
8760494c72 first batch of work towards the VC/BN split: 
- we have a new binary which connects via RPC to the respective BN and has an internal clock - waking it up on every slot
- the BN has a new option called --external-validators and currently in order to have the VC binaries to run we need to pass EXTERNAL_VALIDATORS=yes to make
- factored some code out of beacon_node.nim for easier reuse in validator_api.nim and validator_client.nim
- the VC loads its associated private keys from the datadir for its BN
- most of the validator API calls have been implemented as a stub.
- the VC polls its BN at the start of each epoch - getting a list of all active validators for the current epoch - and then continues to request blocks and sign them with its appropriate validators when necessary
 2020-05-25 16:23:15 +03:00
Dustin Brody
3fa2f70853 update all remaining beacon chain spec references from v0.11.1 and v0.11.2 to v0.11.3 2020-05-20 08:48:18 +00:00
tersec
8432932c11
update to spec v0.11.2 to keep compatible with Schlesi (#1036) 
* update to spec v0.11.2 to keep compatible with Schlesi

* update spec/state_transition_epoch spec references to v0.11.2

* bump other spec refs
 2020-05-18 23:05:27 +00:00
Zahary Karadjov
a739d7e8d6 Adapt SSZ to the latest FastStreams API 2020-05-13 12:18:42 +03:00
Jacek Sieka
fb2e0ddbec
sync fixes (#1005) 
* sync fixes

* fix Status message finalized info
* work around sync starting before initial status exchange
* don't fail block on deposit signature check failure (fixes #989)
* print ForkDigest and Version nicely
* dump incoming blocks
* fix crash when libp2p peer connection is closed
* update chunk size to 16 to work around missing blocks when syncing

* bump libp2p

* bump libp2p

* better deposit skip message
 2020-05-11 18:08:52 +00:00
Jacek Sieka
2449d4b479
cache empty slot state root (#961) 
When replaying state transitions, for the slots that have a block, the
state root is taken from the block. For slots that lack a block, it's
currently calculated using hash_tree_root which is expensive.

Caching the empty slot state roots helps us avoid recalculating this
hash, meaning that for replay, hashes are never calculated. This turns
blocks into fairly lightweight "state-diffs"!

* avoid re-saving state when replaying blocks
* advance empty slots slot-by-slot and save root
* fix sim randomness
* fix sim genesis filename
* introduce `isEpoch` to check if a slot is an epoch slot
 2020-05-03 19:44:04 +02:00
Zahary Karadjov
fdcbfdff05 Pass the test suite with a BeaconState ref type 2020-04-26 13:04:53 +03:00
Jacek Sieka
ed74770451
spec: regulate exceptions (#913) 
* spec: regulate exceptions

* a few more simple raises
 2020-04-22 07:53:02 +02:00
Zahary Karadjov
672f690bf6 Implement most of the v0.11.1 networking changes 2020-04-16 01:01:18 +03:00
Dustin Brody
d559f4ee89 create distinct CommitteeIndex type 2020-04-15 09:59:49 +00:00
Dustin Brody
f49dbf68e6 refactor some Merkle proof generation code outside tests; remove unused variables/functions 2020-04-15 09:59:49 +00:00
tersec
ea59f2ecf2
comments-only changes: update a bunch of v0.10.1 spec references to v0.11.0 and explain rationale for extended validation design in code (#847) 2020-04-01 11:41:39 +00:00
tersec
daabb1b5b2
complete all (non-semantic, comment-only) 0.11.0 -> 0.11.1 beacon chain spec updates, mark all beacon chain v0.11.0 spec references as v0.11.1, and remove now unnecessary/unused UpdateFlags vars from 4 test fixtures (#841) 2020-03-30 23:40:24 +00:00
tersec
f5f939bd31
0.11.1 beacon chain spec update (#836) 
* initial 0.11.1 spec commit; no test regressions and finalizes in eth2_network_simulation

* with BLS 0.10/0.11 available, stop skipping attester slashing, proposer slashing, and voluntary exist operations fixture tests

* switch param orders to group state.{fork, genesis_validators_root}; bump spec/datatypes spec version for network purposes

* mark attestation construction and broadcast and some minimal/mainnet constants as 0.11.1-compatible; remove phase 1 sharding constants from minimal which don't exist in that preset
 2020-03-30 11:31:44 +00:00
tersec
53e4b0a26c
complete (except for get_domain(...)) 0.11.0 beacon chain spec update (#822) 
* complete (except for get_domain(...)) 0.11.0 beacon chain spec update

* mark compute_start_slot_at_epoch(...), is_active_validator(...), compute_signing_root(...), and get_seed(...) as 0.11.0
 2020-03-22 16:03:07 +00:00
Dustin Brody
f7ecd09e95 mark aggregate(...), AttesterSlashing, IndexedAttestation, AttestationData, Deposit, DepositData, VoluntaryExit, BeaconBlock, BeaconBlockHeader, BeaconState, HistoricalBatch, Fork, SigningRoot, SignedVoluntaryExit, SignedBeaconBlock, SignedBeaconBlockHeader, get_current_epoch(...), and get_randao_mix(...) as 0.11.0 2020-03-17 12:06:56 +02:00
Dustin Brody
cae1c0fd1a address beacon_node TODO regarding missing proposer index; mark initiate_validator_exit(...), is_eligible_for_activation_queue(...), Attestation, DepositMessage, BeaconBlockBody, PendingAttestation, Eth1Data, and integer_squareroot(...) as 0.11.0 2020-03-17 12:06:56 +02:00
Dustin Brody
ebcde664e5 mark decrease_balance(...), get_validator_churn_limit(...), network spec version, Checkpoint, compute_epoch_at_slot(...), get_active_validator_indices(...), and get_committee_count_at_slot(...) as 0.11.0 2020-03-17 12:06:56 +02:00