status-im/nimbus-eth2
2
0
Fork 0
mirror of https://github.com/status-im/nimbus-eth2.git synced 2025-01-12 15:24:14 +00:00
Code Issues Projects Releases Wiki Activity
3,055 Commits 106 Branches 116 Tags 245 MiB
Commit Graph

726 Commits

Author SHA1 Message Date
Jacek Sieka
7c0b4d28d2 speed up reward/penalty calculation 
Calculating rewards/penalties is slow due to how we compute sets of
attestations validators then use the sets for inclusion checks, to see
who attested. The dominant function during validated block processing /
epoch processing is hash set building and lookup.

This PR inverts the flow by removing the sets and creating a single
large validator status list, then applying all relevant state
attestations, then updating rewards and penalties.

This provides a 10x speedup to epoch processing which in turn speeds up
both empty slot and block processing - for example, on startup, we
replay all non-finalized blocks to prime fork choice - the same when
validating attestations or replaying states on reorg.
 2020-10-23 19:23:36 +03:00
tersec
a136c2e95a
bump libp2p; integrate pubsub.ValidationResult into extended validation (#1893) 2020-10-20 12:31:20 +00:00
tersec
e87e0ef179
update to 1.0.0-rc.0 message ID; unexport funcs; enable test in mainnet; remove unused subscribe-with-nondummy-handler function; fix gossip received metric (#1884) 2020-10-20 08:54:11 +00:00
tersec
c446eb7cff
update v1.0.0-rc.0 preset spec references (#1891) 
* update ve1.0.0-rc.0 preset spec references

* remove runtime preset ETH1_FOLLOW_DISTANCE from preset files; remove two CI build items to try to keep Travis from timing out
 2020-10-19 13:53:23 +00:00
Jacek Sieka
df43b8aa8b
save some more states after all (#1887) 
Don't save states when replaying history, but do save states when
applying new blocks (!)
 2020-10-18 15:47:39 +00:00
Zahary Karadjov
948d15ea95 Follow-up on #1687 Don't ask for password on invalid keystores 2020-10-16 11:45:39 +03:00
Zahary Karadjov
5f6bdc6709 Store all deposit-derived data in memory 2020-10-15 20:15:51 +03:00
Zahary Karadjov
7a577b2cef More tests for getBlockRange 2020-10-15 20:15:51 +03:00
Zahary Karadjov
2152dc6136 Simplify the mainchain monitor 2020-10-15 20:15:51 +03:00
Zahary Karadjov
4d66914f5a Fix the test suite 2020-10-15 20:15:51 +03:00
Zahary Karadjov
02a3770803 Adds a fuzzing test for the validator client CLI 
Also removes an obsolete way to add validators through the command line
 2020-10-15 20:15:51 +03:00
Zahary Karadjov
8ce0fc3a89 Address #1687 
This addresses the issues by detecting and rejecting keystores with
incorrect PBKDF2 and SCrypt params. It also bumps the version of
nim-json-serialization to include a bugfix for incorrect parsing
of json files featuring comments.
 2020-10-15 20:15:51 +03:00
Jacek Sieka
6b9419e547
fix db growth on attestation processing (#1860) 
It turns out that we often save lots of states in the database that are
the result of empty slot processing only - here, we make sure to only
save a state if a block follows - this fixes several issues:

* empty slot states are not always pruned leading to state database size
explosion
* storing states is (very) slow which slows down processing in general,
so we should only do it when it's likely to be useful
* attestation processing doesn't get stuck on saving random states that
won't appear in the chain history
 2020-10-15 14:28:44 +02:00
tersec
513ba72b9a
add v1.0.0-rc.0 support behind compile-time flag (#1852) 
* add v1.0.0-rc.0 support behind compile-time flag

* keep runtime presets consistent
 2020-10-13 17:21:25 +00:00
tersec
3ee2dd8da4
p2p-interface spec ref bump (except non-updated places) (#1862) 2020-10-12 14:37:14 +00:00
tersec
1994ffe5a0
update 130+ spec references from v0.12.3 to v1.0.0-rc1 (#1854) 2020-10-12 08:59:24 +00:00
tersec
b79e5f8af5
update nim-beacon-chain to nimbus-eth2 in beacon_chain/, ncli/, tests/, and README.md (#1843) 2020-10-08 19:02:05 +00:00
tersec
f08f44b9a2
in exit pool, bundle receive messages into beaconblocks (#1812) 
* in exit pool, filter out already-packaged messages; bundle remaining messages into beaconblocks

* filter messages at block construction time

* allow adding up to intended capacity of buffers, beyond per-block limits

* document rationale/design for filtering mechanism
 2020-10-07 16:57:21 +00:00
Zahary Karadjov
2ea5385aa0 Fix invalid file permissions causilg make eth2_network_sim to fail 2020-10-07 18:12:21 +03:00
Zahary Karadjov
aed291128a Add support for starting from weak subjectivity checkpoints 
Also removes the `genesis.ssz` file stored in the data folder.
The `medalla-fast-sync` target has been adapted to use the new features.
 2020-10-07 09:32:03 +03:00
Zahary Karadjov
78953fd9b5 Address #1729: NFKD Normalization 2020-10-06 15:46:22 +03:00
Mamy Ratsimbazafy
c92d228ec8
Fix non-split attesting protection (#1804) 
* address #1679 - att protection in non-split mode

* AttachedValidator public_key -> pubkey

* Always build with slashing protection

* Remove now unnecessary forwarding impl proc
 2020-10-06 10:51:33 +02:00
cheatfate
dd8b73c85c Remove keypath and set version to 1 for network keystorage. 2020-10-05 22:19:50 +03:00
cheatfate
d2f8cf9386 Fix tests compilation problems. 2020-10-05 22:19:50 +03:00
cheatfate
add22a20e1 Update local_testnet and simulation scripts to use netkey-file and insecure-netkey-password. 
Add more logging
 2020-10-05 22:19:50 +03:00
cheatfate
cec4ad2717 Refactor keystore.nim to include network key storage too. 
Use constant-time hex encoding/decoding procedures.
Add tests for network key storage.
 2020-10-05 22:19:50 +03:00
Zahary Karadjov
f5340998b9 Add a fuzzing test for the beacon node CLI parser 2020-10-02 12:42:32 +03:00
tersec
7eaaab908c
fix output of proposer slashing test fixture (#1780) 
* fix output of proposer slashing test fixture

* run make test

* a few more v0.12.3 spec refs
 2020-09-30 13:12:03 +00:00
Mamy André-Ratsimbazafy
4be37b7c4d
Don't resurrect submodules 2020-09-28 19:17:17 +02:00
mratsim
fe16fba07d Makefile fix for make 3.81 (macOS) 2020-09-28 19:13:36 +02:00
Jacek Sieka
c472d5368b
avoid attestation pool copy in check (#1755) 2020-09-25 19:51:44 +02:00
Mamy Ratsimbazafy
94120ad011
Slash protection fixes (#1757) 
* Address #1698

* Add test for #1699
 2020-09-25 19:39:06 +02:00
tersec
6cf7e837ce
refactor exit pool and gossipsub-subscribe to exit pool message types (#1733) 
* refactor exit pool and gossipsub-subscribe to all exit pool message types

* remove unused loop counter
 2020-09-24 19:05:49 +02:00
tersec
6398a43cc1
update 120+ beacon_chain and validator spec refs from v0.12.2 to v0.12.3 (#1740) 2020-09-24 19:04:10 +02:00
Mamy Ratsimbazafy
4e23b0ef23
Fix https://github.com/status-im/nim-beacon-chain/issues/1701 + raise concerns in keystore secrets protection (#1726) 2020-09-24 07:27:56 +02:00
Eugene Kabanov
654b8d66bf
Peer management (#1707) 
* addPeer() and addPeerNoWait() now returns PeerStatus, not bool.
Minor refactoring of PeerPool.
Fix tests.

* Refactor PeerPool.
Add lenSpace.
Add tests for lenSpace.
PeerPool.add procedures now return different error codes.
Fix SyncManager break/continue problem.
Fix connectWorker break/continue problem.
Refactor connectWorker and discoveryLoop.
Fix incoming/outgoing blocking problem.

* Refactor discovery loop.
Add checkPeer.

* Fix logic and compilation bugs.

* Adjust position of debugging log.

* Fix issue with maximum peers in PeerPool.
Optimize node record decoding.

* fix discoveryLoop.

* Remove aliases and fix tests using aliases.
 2020-09-21 18:02:27 +02:00
tersec
3190c695b0
minimal v0.12.3 update (#1716) 2020-09-21 15:58:35 +00:00
tersec
e106549efe
keep REJECT/IGNORE of messages failing validation for libp2p scoring (#1676) 
* keep REJECT/IGNORE status of messages failing validation for libp2p scoring

* fix test suite
 2020-09-18 13:53:09 +02:00
Jacek Sieka
dcf8a6b05d
improve slot processing speeds (#1670) 
about 40% better slot processing times (with LTO enabled) - these don't
do BLS but are used
heavily during replay (state transition = slot + block transition)

tests using a recent medalla state and advancing it 1000 slots:

```
./ncli slots --preState2:state-302271-3c1dbf19-c1f944bf.ssz --slot:1000
--postState2:xx.ssz
```
pre:

```

All time are ms
Average,       StdDev,          Min,          Max,      Samples,
Test
Validation is turned off meaning that no BLS operations are performed
39.236,        0.000,       39.236,       39.236,            1,
Load state from file
0.049,        0.002,        0.046,        0.063,          968,
Apply slot
256.504,       81.008,      213.471,      591.902,           32,
Apply epoch slot
28.597,        0.000,       28.597,       28.597,            1,
Save state to file
```

cast:
```
All time are ms
Average,       StdDev,          Min,          Max,      Samples,
Test
Validation is turned off meaning that no BLS operations are performed
37.079,        0.000,       37.079,       37.079,            1,
Load state from file
0.042,        0.002,        0.040,        0.090,          968,
Apply slot
215.552,       68.763,      180.155,      500.103,           32,
Apply epoch slot
25.106,        0.000,       25.106,       25.106,            1,
Save state to file
```

cast+rewards:
```
All time are ms
Average,       StdDev,          Min,          Max,      Samples,
Test
Validation is turned off meaning that no BLS operations are performed
40.049,        0.000,       40.049,       40.049,            1,
Load state from file
0.048,        0.001,        0.045,        0.060,          968,
Apply slot
164.981,       76.273,      142.099,      477.868,           32,
Apply epoch slot
28.498,        0.000,       28.498,       28.498,            1,
Save state to file
```

cast+rewards+shr
```
All time are ms
Average,       StdDev,          Min,          Max,      Samples,
Test
Validation is turned off meaning that no BLS operations are performed
12.898,        0.000,       12.898,       12.898,            1,
Load state from file
0.039,        0.002,        0.038,        0.054,          968,
Apply slot
139.971,       68.797,      120.088,      428.844,           32,
Apply epoch slot
24.761,        0.000,       24.761,       24.761,            1,
Save state to file

```
 2020-09-16 20:59:33 +00:00
Mamy Ratsimbazafy
52548f079b
Opt-in Slashing protection + interchange (#1643) 
* Slashing protection + interchange initial commit

* Restrict the when UseSlashingProtection dance in other modules

* Integrate slashing tests in other all_tests

* Add attestation slashing protection support

* Add a message that mention if built with/without slashing protection

* no op the initialization proc

* test slashing protection in Jenkins (temp)

* where to configure NIMFLAGS in Jenkins ...

* Jenkins -> ensure Built with slashing protection

* Add slashing protection complete import

* use Opt.get(otherwise)

* Don't use negation in proc name

* Turn slashing protection on by default
 2020-09-16 13:30:03 +02:00
Eugene Kabanov
6e463257f4
PeerPool fixes. (#1654) 
* Refactor peer_pool.
Fix eth2_network peer counters.
Fix PeerPool do not allow to add more peers when empty space available.

* Remove unused imports.

* Add test for a bug.

* Fix eth2_network disconnect should deletePeer not release.
More PeerPool refactoring.
 2020-09-16 13:00:11 +03:00
Jacek Sieka
c76305f824
fix some todo (#1645) 
* remove some superfluous gcsafes
* remove getTailState (unused)
* don't store old epochrefs in blocks
* document attestation pool a bit
* remove `pcs =` cruft from log
 2020-09-14 14:50:03 +00:00
tersec
aca1a318f2
cleanly close kvstore databases and bump nim-eth (#1630) 
* cleanly close kvstore databases

* close databases for all subcommands and during error conditions
 2020-09-12 05:35:58 +00:00
Jacek Sieka
8a5a261fcd
Quick fix to prune some states, pending smarter state storage (#1624) 
* Quick fix to prune some states, pending smarter state storage

Adverse effects might include slow rewinds - typically the protocol
doesn't ask for pre-finalized states but RPC might

* document issue, add test

* fix cache miss log
 2020-09-11 10:03:50 +02:00
tersec
d0de1a49a3
Fix some warnings and hints and partly revert #1610 (#1615) 
* address some XDeclaredButNotUsed, ConvFromXtoItselfNotNeeded, and UnusedImport hints and warnings

* partly revert #1610
 2020-09-08 11:32:43 +00:00
tersec
3d5f24f14c
stop discarding future epochs; remove a StateCache() construction (#1610) 
* stop discarding non-existent future epochs during epoch state transitions; remove a pointless StateCache() construction in advance_slots()

* update nbench to pass StateCache to process_slots()
 2020-09-07 15:04:33 +00:00
Viktor Kirilov
d9f9949ef0 use a separate process for the private keys (Off by default) - there is a new signing_process binary which loads all validators of the beacon node and the BN dictates through stdin of the signing process what to be signed and when and reads from stdout of the process 2020-09-02 12:47:00 +03:00
Viktor Kirilov
65d7787b1e 50/50 bn/vc split for the validator keys ON by default for the testnet scripts 2020-09-01 16:39:07 +03:00
tersec
ab255662df
bound block quarantine size (#1564) 
* bound block quarantine size

* add additional logging for block quarantining

* re-add quarantine.add() call

* remove pre-finalization blocks; add logging for full quarantine

* clear quarantine on chain reorganization

* update block_sim and tests

* update test_attestation_pool
 2020-08-31 11:00:38 +02:00
Jacek Sieka
fa1621db46
implement clock disparity for attestation validation (#1568) 
This implements disparity, resolving a part of
https://github.com/status-im/nim-beacon-chain/issues/1367

* make BeaconTime a duration for fractional seconds
* factor out attestation/aggregate validation
* simplify recording of queued attestations
* simplify attestation signature check
* fix blocks_received metric
* add some trivial validation tests
* remove unresolved attestation table - attestations for unknown blocks
are dropped instead (cannot verify their signature)
 2020-08-27 09:34:12 +02:00