Commit Graph

1063 Commits

Author SHA1 Message Date
Eugene Kabanov 50c5d47250
Add maximum number of workers (peers used) by SyncManager (default: 10) (#1172)
Refactor and simplification of `sync` procedure.
Fix aggressive looping on excessive recurring failures.
2020-06-14 11:45:53 +02:00
Jacek Sieka 360ebd705f db: compress with snappy
* 3gb vs 12gb for 4000 epochs of witti
* 3-4x sync blocks/sec performance improvement on my FDE SSD drive
* generate less quirky code for primitive types
2020-06-14 11:33:00 +03:00
Dustin Brody 1d1e90adf4 Update 36 spec refs from 0.11.x to 0.12.1 2020-06-12 19:41:03 +00:00
Jacek Sieka 78b767f645
avoid genericAssign for beacon node types (#1166)
* avoid genericAssign for beacon node types

ok, I got fed up of this function messing up cpu measurements - it's so
ridiculously slow, it's sad.

before, while syncing:

```
40,65%  beacon_node_shared_witti_0  [.]
genericAssignAux__U5DxFPRpHCCZDKWQzM9adaw
   9,02%  libc-2.31.so                [.] __memmove_avx_unaligned_erms
   7,07%  beacon_node_shared_witti_0  [.] BIG_384_58_monty
   5,19%  beacon_node_shared_witti_0  [.] BIG_384_58_mul
   2,72%  beacon_node_shared_witti_0  [.] memcpy@plt
   1,18%  [kernel]                    [k] rb_next
   1,17%  beacon_node_shared_witti_0  [.] genericReset
   1,06%  [kernel]                    [k] map_private_extent_buffer
```

after:

```
  24,88%  beacon_node_shared_witti_0  [.] BIG_384_58_monty
  20,29%  beacon_node_shared_witti_0  [.] BIG_384_58_mul
   3,15%  beacon_node_shared_witti_0  [.] BIG_384_58_norm
   2,93%  beacon_node_shared_witti_0  [.] BIG_384_58_add
   2,55%  beacon_node_shared_witti_0  [.] BIG_384_58_sqr
   1,64%  beacon_node_shared_witti_0  [.] BIG_384_58_mod
1,63%  beacon_node_shared_witti_0  [.]
sha256Transform__BJNBQtWr9bJwzqbyfKXd38Q
   1,48%  beacon_node_shared_witti_0  [.] FP_BLS381_add
   1,39%  beacon_node_shared_witti_0  [.] BIG_384_58_sub
   1,33%  beacon_node_shared_witti_0  [.] BIG_384_58_dnorm
   1,14%  beacon_node_shared_witti_0  [.] FP2_BLS381_mul
   1,05%  beacon_node_shared_witti_0  [.] BIG_384_58_cmove
1,05%  beacon_node_shared_witti_0  [.]
get_shuffled_seq__4uncAHNsSG3Pndo5H11U9aQ
```

* better field iteration
2020-06-12 21:10:22 +02:00
Jacek Sieka 42832cefa8
Small fixes (#1165)
* random fixes

* create dump dir on startup
* don't crash on failure to write dump
* fix a few `uint64` instances being used when indexing arrays - this
should be a compile error but isn't due to compiler bugs
* fix standalone test_block_pool compilation
* add signed block processing in ncli

* reuse cache entry instead of allocating a new one

* allow for small clock disparities when validating blocks
2020-06-12 18:43:20 +02:00
tersec c8f24ae3b8
Remove three skipMerkleValidation usages (#1164)
* remove three skipMerkleValidation usages

* remove a couple obsolete comments/TODOs
2020-06-12 18:03:46 +02:00
Kim De Mey 68a8b7d969
Filter discovery nodes on forkId (#1162) 2020-06-12 16:14:18 +02:00
Zahary Karadjov 9363169098 Cosmetic improvement for the statusbar 2020-06-11 17:40:08 +03:00
Zahary Karadjov cf6a869e9e Address some TODO items; Handle start-up before genesis more properly 2020-06-11 17:40:08 +03:00
Zahary Karadjov c773e10c1a Attempt to reduce the risk of dropped network connections during the loading of KeyStores 2020-06-11 17:40:08 +03:00
Zahary Karadjov 25821331c4 More greppable code for the onPeerConnected operation 2020-06-11 17:40:08 +03:00
Zahary Karadjov 811ba9aacd Fix SSZ issues on 32-bit platforms 2020-06-11 17:40:08 +03:00
Zahary Karadjov fdaf419e41 Address review comments 2020-06-11 17:40:08 +03:00
Zahary Karadjov 6e88a07b51 Fix the validator client build 2020-06-11 17:40:08 +03:00
Zahary Karadjov 2acda1c115 Provide a default value for secretsDir (similar to validatorsDir) 2020-06-11 17:40:08 +03:00
Zahary Karadjov a75c632f7a Fixed launch_local_testnet; Renamed validator_keygen to keystore_directories 2020-06-11 17:40:08 +03:00
Zahary Karadjov a8113cf2bc Restore the local sim to a working state 2020-06-11 17:40:08 +03:00
Zahary Karadjov 17343442ea Implement more of the KeyStore spec and integrate it in the beacon node 2020-06-11 17:40:08 +03:00
Eugene Kabanov 1fc9413c48
Fix #1153. (#1160)
Add ability for SyncQueue to recover from unexpected MissingParent.
2020-06-11 16:20:53 +02:00
Jacek Sieka 016cc22173
show peer info on connect (#1155) 2020-06-11 07:14:26 +02:00
Viktor Kirilov e7febc2e2b can finalize successfully with external VCs! + some other cleanup 2020-06-10 13:50:50 +03:00
Viktor Kirilov 06873abde3 fixed build 2020-06-10 13:50:50 +03:00
Viktor Kirilov 3bae40ae91 - extracted the commands to run a VC into a separate run_validator.sh script
- each BN gets half of its previous validators as inProcess and the other half goes to the respective VC for that BN - using separate data dirs where the keys are copied
    - also removed a few command line options which are no longer necessary
- block proposals originating from a VC are propagated from one BN to the rest properly
- other cleanup & moving code back to  since it is no longer used elsewhere
2020-06-10 13:50:50 +03:00
Mamy Ratsimbazafy ce897fe83f
[Split fork choice PR] Derisk-ed attestation checks changes (#1154)
* Derisked attestation pool improvements

* tune down frequent logs

* VoteTracker logging
2020-06-10 08:58:12 +02:00
Dustin Brody a9f73ac8fb poke 2020-06-09 18:39:05 +00:00
Dustin Brody 8f7541959a allow compile-time switching between v0.11.3 and v0.12.1 2020-06-09 18:39:05 +00:00
Dustin Brody 36f15a3688 complete re-implmentation of reasonable asymptotic-complexity algorithms from 0.11.3 2020-06-09 18:39:05 +00:00
Dustin Brody 35f67cec60 remove the scaffolding get_empty_per_epoch_cache() calls and the pointless construction and return/discarding of zero-filled arrays 2020-06-09 18:39:05 +00:00
Dustin Brody 5582ecc1f4 remove some debugging code 2020-06-09 18:39:05 +00:00
Dustin Brody cc13153fc6 initial 0.12.1 beacon chain spec 2020-06-09 18:39:05 +00:00
Jacek Sieka 1e71388093
Include finalization, head and time (in that order) in standard bar (#1142)
These are the 3 most important things in the life of a beacon-node -
finalization checkpoint for security, head + time for seeing that we're
synced and working.

The order given is the expected increasing order that things should be
moving in - finalization lags head, and head lags clock - the structure
helps the user remember, but also see the pattern where time is getting
out of sync with head.

A validator would also want to know when the next action is going to be
(ie next attestation / block production) but that's for another day.
2020-06-08 18:04:11 +02:00
Eugene Kabanov 040e38529a
Fix #1140. (#1143)
SeenTable is now able to hold peers with different timeout values.
2020-06-08 18:02:50 +02:00
tersec a50415a719
temporary workaround for witti attestation dropping (#1131) 2020-06-08 09:22:31 +02:00
Eugene Kabanov 3ce98d5bca
Add checks for penalties which are not applied immediately. (#1139)
Change default maxHeadAge value to 1 epoch.
Set zero-point at the SyncQueue's initialization.
Remove annoying logs in runDiscoveryLoop.
2020-06-07 17:36:24 +02:00
Jacek Sieka ea8f96d284
ncli_db: allow saving states by root (#1136)
also dump state+block when validation fails
2020-06-06 13:26:19 +02:00
Jacek Sieka bb698a7e16
lots of peers (#1130) 2020-06-05 22:25:01 +02:00
Zahary Karadjov 4f1c4687e4
Fix the inspector build 2020-06-05 20:06:24 +03:00
Zahary Karadjov 0c78fc39e7
Use the latest LibP2P 2020-06-05 19:34:12 +03:00
Zahary Karadjov 32bc4d9096 Use {.push raises.} in spec/datatypes
This was an attempt to trigger a previously reported crash, but
everything turned out to be fine.
2020-06-05 19:33:30 +03:00
Dustin Brody 74dc2fffa6 3x blocksim speedup by using EpochRef in attestation pool addResolved(...) 2020-06-05 13:02:35 +00:00
Jacek Sieka 68b5638da4 allow non-power-of-2 limits in hashlist
fixes "make eth2_network_simulation"

a bit sad: no test coverage except against our own tests
2020-06-05 14:51:22 +03:00
Dustin Brody 3cb7896bab 12x speedup on state sim with 100k validators sans BLS by caching get_beacon_proposer_index(...) 2020-06-04 17:07:51 +00:00
Jacek Sieka bcbfa736c9
format ErrorMsg messages reasonably (#1109) 2020-06-04 08:19:25 +02:00
Jacek Sieka 56ffb696be
reorder ssz (#1099)
* reorder ssz

* split into hash_trees and ssz_serialization, roughly, for hashing and
IO
* move bitseqs into ssz (from stew)
* clean up imports

* docs, imports
2020-06-03 15:52:02 +02:00
cheatfate 405e9db199
Fix problem of good peers is also logged as timed out. 2020-06-03 13:48:01 +03:00
cheatfate 12e28a1fa9 Add proper concurrent connections.
Add SeenTable to avoid continuous attempts to dead peers.
Refactor onSecond.
Block backward sync while forward sync is working.
SyncManager now checks responses according corresponding requests + tests.
SyncManager now watching for not progressing local_head_slot and resets SyncQueue.
2020-06-03 12:53:57 +03:00
Jacek Sieka 07393c8de6
Ssz cache (#1084)
* wip: cache

* cache lists and arrays of complex objects (5x block processing speed
on ncli_db)

trivial baseline cache that stores tree in flat memory structure

* support array of uint64

* work around type issues

* more type compiler bug workarounds

* cache balances, more type fixes

* index type

* ncli_db: add validation flag, better ux

* int64 fixes

* test fix

* "oops"

```
647.913,        0.000,      647.913,      647.913,            1,
Initialize DB
0.540,        0.402,        0.340,        9.451,          619,
Load block from database
40.268,        0.000,       40.268,       40.268,            1,
Load state from database
0.498,        0.150,        0.343,        0.930,          596,
Apply block
3.548,       11.005,        0.729,       54.022,           23,
Apply epoch block
```

* support all basic types

* cleanups

* a few more cleanups
2020-06-03 07:57:17 +02:00
tersec b7bb62d48b
use shuffled validator cache in more places; cleanups (#1095) 2020-06-03 07:42:08 +02:00
Jacek Sieka 1d9a58f1e8
a few more cleanups 2020-06-02 22:10:45 +02:00
kdeme 06f025b228 Add timeout to switch.connect 2020-06-02 23:06:11 +03:00