Commit Graph

1160 Commits

Author SHA1 Message Date
Jacek Sieka 78b767f645
avoid genericAssign for beacon node types (#1166)
* avoid genericAssign for beacon node types

ok, I got fed up of this function messing up cpu measurements - it's so
ridiculously slow, it's sad.

before, while syncing:

```
40,65%  beacon_node_shared_witti_0  [.]
genericAssignAux__U5DxFPRpHCCZDKWQzM9adaw
   9,02%  libc-2.31.so                [.] __memmove_avx_unaligned_erms
   7,07%  beacon_node_shared_witti_0  [.] BIG_384_58_monty
   5,19%  beacon_node_shared_witti_0  [.] BIG_384_58_mul
   2,72%  beacon_node_shared_witti_0  [.] memcpy@plt
   1,18%  [kernel]                    [k] rb_next
   1,17%  beacon_node_shared_witti_0  [.] genericReset
   1,06%  [kernel]                    [k] map_private_extent_buffer
```

after:

```
  24,88%  beacon_node_shared_witti_0  [.] BIG_384_58_monty
  20,29%  beacon_node_shared_witti_0  [.] BIG_384_58_mul
   3,15%  beacon_node_shared_witti_0  [.] BIG_384_58_norm
   2,93%  beacon_node_shared_witti_0  [.] BIG_384_58_add
   2,55%  beacon_node_shared_witti_0  [.] BIG_384_58_sqr
   1,64%  beacon_node_shared_witti_0  [.] BIG_384_58_mod
1,63%  beacon_node_shared_witti_0  [.]
sha256Transform__BJNBQtWr9bJwzqbyfKXd38Q
   1,48%  beacon_node_shared_witti_0  [.] FP_BLS381_add
   1,39%  beacon_node_shared_witti_0  [.] BIG_384_58_sub
   1,33%  beacon_node_shared_witti_0  [.] BIG_384_58_dnorm
   1,14%  beacon_node_shared_witti_0  [.] FP2_BLS381_mul
   1,05%  beacon_node_shared_witti_0  [.] BIG_384_58_cmove
1,05%  beacon_node_shared_witti_0  [.]
get_shuffled_seq__4uncAHNsSG3Pndo5H11U9aQ
```

* better field iteration
2020-06-12 21:10:22 +02:00
Jacek Sieka 42832cefa8
Small fixes (#1165)
* random fixes

* create dump dir on startup
* don't crash on failure to write dump
* fix a few `uint64` instances being used when indexing arrays - this
should be a compile error but isn't due to compiler bugs
* fix standalone test_block_pool compilation
* add signed block processing in ncli

* reuse cache entry instead of allocating a new one

* allow for small clock disparities when validating blocks
2020-06-12 18:43:20 +02:00
tersec c8f24ae3b8
Remove three skipMerkleValidation usages (#1164)
* remove three skipMerkleValidation usages

* remove a couple obsolete comments/TODOs
2020-06-12 18:03:46 +02:00
Kim De Mey 68a8b7d969
Filter discovery nodes on forkId (#1162) 2020-06-12 16:14:18 +02:00
Zahary Karadjov 9363169098 Cosmetic improvement for the statusbar 2020-06-11 17:40:08 +03:00
Zahary Karadjov cf6a869e9e Address some TODO items; Handle start-up before genesis more properly 2020-06-11 17:40:08 +03:00
Zahary Karadjov c773e10c1a Attempt to reduce the risk of dropped network connections during the loading of KeyStores 2020-06-11 17:40:08 +03:00
Zahary Karadjov 25821331c4 More greppable code for the onPeerConnected operation 2020-06-11 17:40:08 +03:00
Zahary Karadjov 811ba9aacd Fix SSZ issues on 32-bit platforms 2020-06-11 17:40:08 +03:00
Zahary Karadjov fdaf419e41 Address review comments 2020-06-11 17:40:08 +03:00
Zahary Karadjov 6e88a07b51 Fix the validator client build 2020-06-11 17:40:08 +03:00
Zahary Karadjov 2acda1c115 Provide a default value for secretsDir (similar to validatorsDir) 2020-06-11 17:40:08 +03:00
Zahary Karadjov a75c632f7a Fixed launch_local_testnet; Renamed validator_keygen to keystore_directories 2020-06-11 17:40:08 +03:00
Zahary Karadjov a8113cf2bc Restore the local sim to a working state 2020-06-11 17:40:08 +03:00
Zahary Karadjov 17343442ea Implement more of the KeyStore spec and integrate it in the beacon node 2020-06-11 17:40:08 +03:00
Eugene Kabanov 1fc9413c48
Fix #1153. (#1160)
Add ability for SyncQueue to recover from unexpected MissingParent.
2020-06-11 16:20:53 +02:00
Jacek Sieka 016cc22173
show peer info on connect (#1155) 2020-06-11 07:14:26 +02:00
Viktor Kirilov e7febc2e2b can finalize successfully with external VCs! + some other cleanup 2020-06-10 13:50:50 +03:00
Viktor Kirilov 06873abde3 fixed build 2020-06-10 13:50:50 +03:00
Viktor Kirilov 3bae40ae91 - extracted the commands to run a VC into a separate run_validator.sh script
- each BN gets half of its previous validators as inProcess and the other half goes to the respective VC for that BN - using separate data dirs where the keys are copied
    - also removed a few command line options which are no longer necessary
- block proposals originating from a VC are propagated from one BN to the rest properly
- other cleanup & moving code back to  since it is no longer used elsewhere
2020-06-10 13:50:50 +03:00
Mamy Ratsimbazafy ce897fe83f
[Split fork choice PR] Derisk-ed attestation checks changes (#1154)
* Derisked attestation pool improvements

* tune down frequent logs

* VoteTracker logging
2020-06-10 08:58:12 +02:00
Dustin Brody a9f73ac8fb poke 2020-06-09 18:39:05 +00:00
Dustin Brody 8f7541959a allow compile-time switching between v0.11.3 and v0.12.1 2020-06-09 18:39:05 +00:00
Dustin Brody 36f15a3688 complete re-implmentation of reasonable asymptotic-complexity algorithms from 0.11.3 2020-06-09 18:39:05 +00:00
Dustin Brody 35f67cec60 remove the scaffolding get_empty_per_epoch_cache() calls and the pointless construction and return/discarding of zero-filled arrays 2020-06-09 18:39:05 +00:00
Dustin Brody 5582ecc1f4 remove some debugging code 2020-06-09 18:39:05 +00:00
Dustin Brody cc13153fc6 initial 0.12.1 beacon chain spec 2020-06-09 18:39:05 +00:00
Jacek Sieka 1e71388093
Include finalization, head and time (in that order) in standard bar (#1142)
These are the 3 most important things in the life of a beacon-node -
finalization checkpoint for security, head + time for seeing that we're
synced and working.

The order given is the expected increasing order that things should be
moving in - finalization lags head, and head lags clock - the structure
helps the user remember, but also see the pattern where time is getting
out of sync with head.

A validator would also want to know when the next action is going to be
(ie next attestation / block production) but that's for another day.
2020-06-08 18:04:11 +02:00
Eugene Kabanov 040e38529a
Fix #1140. (#1143)
SeenTable is now able to hold peers with different timeout values.
2020-06-08 18:02:50 +02:00
tersec a50415a719
temporary workaround for witti attestation dropping (#1131) 2020-06-08 09:22:31 +02:00
Eugene Kabanov 3ce98d5bca
Add checks for penalties which are not applied immediately. (#1139)
Change default maxHeadAge value to 1 epoch.
Set zero-point at the SyncQueue's initialization.
Remove annoying logs in runDiscoveryLoop.
2020-06-07 17:36:24 +02:00
Jacek Sieka ea8f96d284
ncli_db: allow saving states by root (#1136)
also dump state+block when validation fails
2020-06-06 13:26:19 +02:00
Jacek Sieka bb698a7e16
lots of peers (#1130) 2020-06-05 22:25:01 +02:00
Zahary Karadjov 4f1c4687e4
Fix the inspector build 2020-06-05 20:06:24 +03:00
Zahary Karadjov 0c78fc39e7
Use the latest LibP2P 2020-06-05 19:34:12 +03:00
Zahary Karadjov 32bc4d9096 Use {.push raises.} in spec/datatypes
This was an attempt to trigger a previously reported crash, but
everything turned out to be fine.
2020-06-05 19:33:30 +03:00
Dustin Brody 74dc2fffa6 3x blocksim speedup by using EpochRef in attestation pool addResolved(...) 2020-06-05 13:02:35 +00:00
Jacek Sieka 68b5638da4 allow non-power-of-2 limits in hashlist
fixes "make eth2_network_simulation"

a bit sad: no test coverage except against our own tests
2020-06-05 14:51:22 +03:00
Dustin Brody 3cb7896bab 12x speedup on state sim with 100k validators sans BLS by caching get_beacon_proposer_index(...) 2020-06-04 17:07:51 +00:00
Jacek Sieka bcbfa736c9
format ErrorMsg messages reasonably (#1109) 2020-06-04 08:19:25 +02:00
Jacek Sieka 56ffb696be
reorder ssz (#1099)
* reorder ssz

* split into hash_trees and ssz_serialization, roughly, for hashing and
IO
* move bitseqs into ssz (from stew)
* clean up imports

* docs, imports
2020-06-03 15:52:02 +02:00
cheatfate 405e9db199
Fix problem of good peers is also logged as timed out. 2020-06-03 13:48:01 +03:00
cheatfate 12e28a1fa9 Add proper concurrent connections.
Add SeenTable to avoid continuous attempts to dead peers.
Refactor onSecond.
Block backward sync while forward sync is working.
SyncManager now checks responses according corresponding requests + tests.
SyncManager now watching for not progressing local_head_slot and resets SyncQueue.
2020-06-03 12:53:57 +03:00
Jacek Sieka 07393c8de6
Ssz cache (#1084)
* wip: cache

* cache lists and arrays of complex objects (5x block processing speed
on ncli_db)

trivial baseline cache that stores tree in flat memory structure

* support array of uint64

* work around type issues

* more type compiler bug workarounds

* cache balances, more type fixes

* index type

* ncli_db: add validation flag, better ux

* int64 fixes

* test fix

* "oops"

```
647.913,        0.000,      647.913,      647.913,            1,
Initialize DB
0.540,        0.402,        0.340,        9.451,          619,
Load block from database
40.268,        0.000,       40.268,       40.268,            1,
Load state from database
0.498,        0.150,        0.343,        0.930,          596,
Apply block
3.548,       11.005,        0.729,       54.022,           23,
Apply epoch block
```

* support all basic types

* cleanups

* a few more cleanups
2020-06-03 07:57:17 +02:00
tersec b7bb62d48b
use shuffled validator cache in more places; cleanups (#1095) 2020-06-03 07:42:08 +02:00
Jacek Sieka 1d9a58f1e8
a few more cleanups 2020-06-02 22:10:45 +02:00
kdeme 06f025b228 Add timeout to switch.connect 2020-06-02 23:06:11 +03:00
Jacek Sieka 7e881a4c09
cleanups 2020-06-02 17:17:07 +02:00
Jacek Sieka 872d7ff493
support all basic types 2020-06-02 11:44:51 +02:00
Jacek Sieka 00acf4f7b2
"oops"
```
647.913,        0.000,      647.913,      647.913,            1,
Initialize DB
0.540,        0.402,        0.340,        9.451,          619,
Load block from database
40.268,        0.000,       40.268,       40.268,            1,
Load state from database
0.498,        0.150,        0.343,        0.930,          596,
Apply block
3.548,       11.005,        0.729,       54.022,           23,
Apply epoch block
```
2020-06-01 22:37:26 +02:00
Dustin Brody 1a6b83a15f cleanups 2020-06-01 19:45:21 +00:00
Jacek Sieka 405f460f9b
int64 fixes 2020-06-01 17:35:42 +02:00
Zahary Karadjov ef90a45509
Bump testutils 2020-06-01 16:58:28 +03:00
Jacek Sieka 2316ac15c3
index type 2020-06-01 14:52:52 +02:00
Jacek Sieka fec01829c4
cache balances, more type fixes 2020-06-01 14:52:52 +02:00
Jacek Sieka be92842944
more type compiler bug workarounds 2020-06-01 14:52:52 +02:00
Jacek Sieka a0b8b818f3
work around type issues 2020-06-01 14:52:52 +02:00
Jacek Sieka 4fc9a84c57
support array of uint64 2020-06-01 14:52:52 +02:00
Jacek Sieka f1dcee2749
wip: cache
* cache lists and arrays of complex objects (5x block processing speed
on ncli_db)

trivial baseline cache that stores tree in flat memory structure
2020-06-01 14:52:50 +02:00
tersec a327e8581b
switch state transition caching to match EpochRef (#1089)
* switch state transition caching usage to shuffled active validator indices to match EpochRef

* refactor the EpochRef -> StateCache transformation; elide pointless mapIt

* limit state passed between get_beacon_committee(...) and compute_committee(...)

* tweaks
2020-06-01 09:44:50 +02:00
Zahary Karadjov 0bcdabfcdf Detect another invalid SSZ input found through fuzzing
The first offset of an SSZ object should always have a fixed constant
value. Otherwise, some unused bytes may appear between the fixed portion
and the dynamic portion.

Please note that this fix shutds down the minimal forward compatibility
currently supported by the SSZ format (and thus, the expected behavior
must be clarified in the SSZ spec).
2020-05-30 14:47:35 +03:00
Ștefan Talpalaru a90b0dd197
Merge pull request #1077 from status-im/timeout
Eth2Node.stop(): trace msg on timeout
2020-05-29 16:30:13 +02:00
Zed c570abf602 Check iv length for decryption 2020-05-29 16:38:55 +03:00
Zed 80ce960afa Minor improvements 2020-05-29 16:38:55 +03:00
Zed 49c083d5b7 Catch all exceptions 2020-05-29 16:38:55 +03:00
Zed 14ad100b45 Address comments 2020-05-29 16:38:55 +03:00
Zed 8496e20a78 Implement EIP 2335 compliant keystore
Closes #1024
2020-05-29 16:38:55 +03:00
Kim De Mey e33c8d9067
Bump nim-eth and accompanying discv5 cleanup (#1081) 2020-05-29 12:03:29 +02:00
tersec b5f45db5e9
keep cache of per-epoch items in block pool (#1068)
* plumbing between block pool and state transition functions around active validator indices and committees

* have shared epochrefs followed by blockref tree while allowing for skipped slots

* factor out the epoch info extraction; document how the EpochRef follows forks
2020-05-29 08:10:20 +02:00
Jacek Sieka 061be037d1 ncli_db: database tool
includes a benchmark tool for now
2020-05-28 17:43:02 +00:00
Zahary Karadjov 0493737d7a Remove ssz.loadFile 2020-05-28 19:15:16 +03:00
Zahary Karadjov 1fd87a2255 Remove SszReader.maxObjectSize 2020-05-28 19:15:16 +03:00
Zahary Karadjov d4fa2c8913 Avoid an unnecessary copy in readSszValue 2020-05-28 19:15:16 +03:00
Jacek Sieka 693bc15919 avoid some RVO bugs 2020-05-28 19:15:16 +03:00
Jacek Sieka f53b55cbe0 SSZ cleanup
* be stricter about SSZ length prefix
* compute zeroHash list at compile time
* remove SSZ schema stuff
* move SSZ navigation to ncli
* cleanup a few leftover openArray uses
2020-05-28 19:15:16 +03:00
Viktor Kirilov cefd525ab3 more work on the BN/VC split
- fixed comments from the last review
- getting more data VIA RPC, moved some code back into the BN only
- attestation duties being requested as well
2020-05-28 13:01:42 +03:00
Jacek Sieka 18eca263b1
avoid stack allocations in some tests (#1079) 2020-05-28 10:28:14 +02:00
Eugene Kabanov 21131e629b
Sync freeze fixes. (#1072)
* Add ability to reset state of sync manager.
Fix bug when sync got stuck on `zero-point` reset.
Fix bug when sync got stuck when some of the workers waiting for failing one.

* Remove debugging comments and imports.

* Remove not used pendingLock.
2020-05-28 07:02:28 +02:00
Ștefan Talpalaru 273a912ae0
Eth2Node.stop(): trace msg on timeout 2020-05-28 03:14:01 +02:00
kdeme 1ec7d6e168 Quickfix for toEnode that fails 2020-05-27 13:49:18 +03:00
Zahary Karadjov 28128f4d2f Add a handler for the Goodbye message
The lack of body of `goodbye` in sync_protocol.nim was preventing
the respective LibP2P protocol to be mounted and advertised on the
network.

Adding a body fixes that, but I've also made some changes in the
P2P protocol codegen that will prevent the issue from happening
again (no body is now considered the equivalent of having an empty
body).
2020-05-26 22:17:26 +03:00
Jacek Sieka 0d54073ec9 properly work around nim type bug in ssz
the cast worked around the bug at compile time by means of casting, but
introduced a runtime error, because the pre-cast type was still being
used during deserialization
2020-05-26 14:12:42 +03:00
Dustin Brody 0929d90d93 unexport candidate_chains.init; some spec version bumps 2020-05-26 05:06:37 +00:00
Zahary Karadjov b6c9de968f
Fix some build problems and warnings 2020-05-25 20:33:03 +03:00
Viktor Kirilov 8760494c72 first batch of work towards the VC/BN split:
- we have a new binary which connects via RPC to the respective BN and has an internal clock - waking it up on every slot
- the BN has a new option called --external-validators and currently in order to have the VC binaries to run we need to pass EXTERNAL_VALIDATORS=yes to make
- factored some code out of beacon_node.nim for easier reuse in validator_api.nim and validator_client.nim
- the VC loads its associated private keys from the datadir for its BN
- most of the validator API calls have been implemented as a stub.
- the VC polls its BN at the start of each epoch - getting a list of all active validators for the current epoch - and then continues to request blocks and sign them with its appropriate validators when necessary
2020-05-25 16:23:15 +03:00
zah 626e51a91b [skip ci] Address core review comments 2020-05-24 19:00:34 +03:00
Zahary Karadjov 4a25bec2b5 Quick and dirty fix for the inspector build 2020-05-24 19:00:34 +03:00
Zahary Karadjov 833f19e942 Reform the networking layer in order to handle the new stricter SSZ API 2020-05-24 19:00:34 +03:00
Zahary Karadjov a8003e7e38 More complete integration of the List type; Detect more invalid inputs; Enable more tests 2020-05-24 19:00:34 +03:00
Zahary Karadjov accd5fe954 Don't use StackArray in ssz; Drop the support for strings 2020-05-24 19:00:34 +03:00
Dustin Brody 74e35c464f halt processing if one of the operation functions fails (failed deposits due to BLS signature issues return as not-failed, so it remains consistant with that 2020-05-24 08:20:13 +00:00
Dustin Brody 67ba11cc1a use template to more cleanly implement for_ops 2020-05-24 08:20:13 +00:00
Dustin Brody 95ee0218db switch too-strict/against-spec operations processing to use spec-based process_operations 2020-05-24 08:20:13 +00:00
Jacek Sieka f06df1cea6 remove some copies
* in makeBeaconBlock - use rollback instead
* in tests - this helps state_sim give more accurate data and makes it
30% faster
* fix some usages of raw BeaconState
2020-05-22 17:15:35 +00:00
Jacek Sieka 681b399e2a
avoid memory alloc when padding (#1058)
also avoid stack state copy in test
2020-05-22 10:31:19 +02:00
tersec 80b8416305
skip skipped constant sanity checks also from markdown reports' perspectives (#1053)
* update more spec refs in beacon_chain/spec/presets; skip skipped constant sanity checks also from markdown reports' perspectives

* mark skipped as skipped in markdown
2020-05-21 19:56:09 +02:00
Jacek Sieka 7fbb8c0bc2
return block result details (#1049) 2020-05-21 19:08:31 +02:00
Ștefan Talpalaru b2193f1b8f
Eth2Node.stop(): 5s timeout 2020-05-21 00:06:01 +02:00
Jacek Sieka a38eddcaac
remove ssz stint support (#1046) 2020-05-20 19:05:22 +02:00
Dustin Brody b9817867f5 audit & update spec references for some slashing and rewards and penalty functions; check and update all remaining v0.11.0 spec references to v0.11.3 [skip ci] 2020-05-20 14:57:08 +00:00
Jacek Sieka 23daa966be
better deserialization log 2020-05-20 15:41:02 +02:00
Jacek Sieka 30f84224a4 create dumpdir in beaconnode binary 2020-05-20 11:03:24 +00:00
Dustin Brody 3fa2f70853 update all remaining beacon chain spec references from v0.11.1 and v0.11.2 to v0.11.3 2020-05-20 08:48:18 +00:00
Jacek Sieka 418888f1d0
use ethkeys to parse enr key 2020-05-20 07:35:34 +02:00
Ștefan Talpalaru 383b22795c
bump submodules (#1043) 2020-05-20 06:57:39 +02:00
Ștefan Talpalaru c4462af4ab
beacon_node: graceful shutdown (#1033)
* beacon_node: graceful shutdown

* separate BeaconNodeStatus and BeaconNode instances
2020-05-19 20:57:35 +02:00
Dustin Brody 6c941b1ff7 spec v0.11.3 update 2020-05-19 17:16:55 +00:00
Mamy Ratsimbazafy c014f0b301
Split quarantine (#1038)
* split blockpool into hotDB and Quarantine

* Rename hotdb -> dag/candidate chains
2020-05-19 16:18:07 +02:00
Zahary Karadjov a99977b772 Progress towards #991; Enable the distinct ssz.List type 2020-05-19 15:53:03 +03:00
Eugene Kabanov ea95021073
Fix sync issues. (#1035)
* Fix sync issues.

* Add documentation about zero-point.
Add more comments about syncing loops.
Change to 4 blocks per request.
2020-05-19 14:08:50 +02:00
Jacek Sieka 4359147efc
fix snappy debug message 2020-05-19 09:23:17 +02:00
tersec 8432932c11
update to spec v0.11.2 to keep compatible with Schlesi (#1036)
* update to spec v0.11.2 to keep compatible with Schlesi

* update spec/state_transition_epoch spec references to v0.11.2

* bump other spec refs
2020-05-18 23:05:27 +00:00
Dmitriy Ryajov 0649d47df0 use proper transport flags 2020-05-18 21:51:03 +00:00
Jacek Sieka a1689f2aaf error when snappy frame is short 2020-05-18 21:15:04 +00:00
Jacek Sieka 6be7d64e8c
bump libp2p (#1031) 2020-05-18 10:11:21 +02:00
Zahary Karadjov 24a17f5814 Fix an RPC error in Lighthouse triggered by the getMetadata message 2020-05-16 09:56:13 +03:00
tersec 74db0f3c8d
fix some XDeclaredButNotUsed hints (#1027) 2020-05-15 14:41:00 +02:00
Mamy Ratsimbazafy 1916e1d2da
Revive nbench (#1022)
* Fix nbench compilation with HashedBeaconState

* Add nbench to tooling

* use newClone - fix 265e01e404 (r425198575)

* Detail advance_slot and hashTreeRoot

* Report throughput

* Fallback for ARM

* windows does not support inline ASM
2020-05-15 00:54:10 +02:00
Dustin Brody 2ddc507e5b update datatypes v0.11.1 spec references to v0.11.2 2020-05-14 16:33:11 +00:00
Dustin Brody 2d691b2e14 bump spec ref versions 2020-05-14 16:41:46 +03:00
Dustin Brody 2db99545cd allow inspector tool to log interop attestations 2020-05-14 16:41:46 +03:00
Dustin Brody 57519bebac remove some unused imports, add tests for pubsub topics, and subscribe to interop attestations 2020-05-14 16:41:46 +03:00
Jacek Sieka 3c98f18d47 document snappy decompression assumptions 2020-05-14 16:41:19 +03:00
Jacek Sieka a605c7244e simplify libp2p snappy
* handle a few more exceptions gracefully (in libp2p also)
* unify libp2p varint parsing
* decompress directly into seq
* avoid seq slice
* stop oversized snappy processing earlier (lowers risk)
2020-05-14 16:41:19 +03:00
Viktor Kirilov 926d1c546e updated news, also no longer allowing event handlers to raise anything other than a Defect in the Web3 API 2020-05-14 14:10:26 +03:00
Diederik Loerakker be806e6801
API fixes - minimal changes to HTTP RPC for Eth2Stats support (#1017)
* add api for eth2stats

* fix chain head rpc
2020-05-14 11:18:08 +02:00
Zahary Karadjov 75c1c6a95c Enable Snappy by default (using LibP2P steams for now)
This refactors the newly added Snappy streaming back-ends trying to
make them more similar and to reduce the code duplication to a minimum.
2020-05-13 12:18:42 +03:00
Zahary Karadjov f055fad08a Make the Snappy FastStreams integration optional by duplicating it for LibP2P streams 2020-05-13 12:18:42 +03:00
Zahary Karadjov 3ecb197635 Memory-optimal hash_tree_root
The work on this was started last week while I was waiting
for a decision on the "Async Snappy" PR. It was prompted by
a failing test in the test suite, where the HashingStream
was inserting some incorrectly padded chunks that affected
the result of `hash_tree_root`. Instead of working around
the problem in the HashingStream, I've decided to implement
a planned optimisation that allows us to remove the hashing
stream altogether.

With the optimisation in place, `hash_tree_root` will now
use only stack memory and only the precise amount neccesary
to build the chunks-merging tree.
2020-05-13 12:18:42 +03:00
Zahary Karadjov 7e846a0bce Implement sszSize 2020-05-13 12:18:42 +03:00
Zahary Karadjov 15f0153441 Cosmetic improvements 2020-05-13 12:18:42 +03:00
Zahary Karadjov e8a15aa0ce Document the SSZ hashing stream 2020-05-13 12:18:42 +03:00
Zahary Karadjov 9538b60704 Integrate the async Snappy implementation 2020-05-13 12:18:42 +03:00
Zahary Karadjov a739d7e8d6 Adapt SSZ to the latest FastStreams API 2020-05-13 12:18:42 +03:00
tersec 9ad05d44e4
reintroduce finalization verification to beacon node, because otherwise the consensus layer can get detached without noticing it (#1013) 2020-05-13 08:36:33 +00:00
Eugene Kabanov da0b1a4993
Fix status handling. (#1008)
* Fix status handling.
Add log map of received blocks.

* Fix review comments.
Fix UnusedImport in sync_protocol.nim
2020-05-13 08:37:58 +02:00
Dustin Brody a58d45b737
remove potentially GC leak-inducing acyclicity tag 2020-05-12 20:20:24 +02:00
tersec ba1d7e2ed4
switch state cache to use ref statedata objects to limit memory usage (#1007)
* switch state cache to use ref statedata objects to limit memory usage

* more directly initialize ref StateData

* use HashedBeaconState instead of StateData to try to fix memory leak

* switch cache to seq[ref HashedBeaconState]

* remove unused import

Co-authored-by: Ștefan Talpalaru <stefantalpalaru@yahoo.com>
2020-05-12 16:26:58 +00:00
Jacek Sieka fb2e0ddbec
sync fixes (#1005)
* sync fixes

* fix Status message finalized info
* work around sync starting before initial status exchange
* don't fail block on deposit signature check failure (fixes #989)
* print ForkDigest and Version nicely
* dump incoming blocks
* fix crash when libp2p peer connection is closed
* update chunk size to 16 to work around missing blocks when syncing

* bump libp2p

* bump libp2p

* better deposit skip message
2020-05-11 18:08:52 +00:00
Ștefan Talpalaru a7a50824a1
more metrics (#1004) 2020-05-11 06:25:49 +00:00
tersec c498103b2f
quick/minimal mitigation of beacon_node memory usage resulting from 2*Table.defaultInitialSize pointless BeaconState objects in block pool state cache (#1002) 2020-05-10 16:31:55 +00:00
Jacek Sieka 592d6f65c3
bump (#1001) 2020-05-09 16:47:14 +02:00
Eugene Kabanov 55dfcc6783
Inspector native version. (#995) 2020-05-09 14:18:58 +00:00
tersec 093d298b2b
Increase finalization and finalization checking robustness (#990)
* fix some warnings related to beacon_node splitting; reimplement finalization verification more robustly; improve attestation pool block selection logic

* re-add missing import

* whitelist allowed state transition flags and make rollback/restore naming more consistent

* restore usage of update flags passed into skipAndUpdateState(...) in addition to the potential verifyFinalization flag

* switch rest of rollback -> restore
2020-05-09 12:43:15 +00:00
Ștefan Talpalaru c572f61129
bump vendor/nim-metrics 2020-05-09 01:13:57 +02:00
Ștefan Talpalaru fc9d6bc1b3
enable SO_REUSEADDR in libp2p transports 2020-05-08 21:50:20 +02:00
Jacek Sieka c1aff83b8b
decrease sync chunk size (#988)
While mainnet performance is being worked on, this helps with the flow.
2020-05-08 08:17:40 +02:00
Eugene Kabanov 3ffd0d7216
Fix gossip messages not delivered. (#983) 2020-05-06 15:42:59 +00:00
Mamy Ratsimbazafy 621c2e38a6
Separate validator duties from the Beacon Node (#976)
* Separate validator duties from the Beacon Node

* remove duplicate MaxEmptySlotCOunt

* imports

* have beacnde properly compile
2020-05-06 13:23:45 +00:00
Jacek Sieka a56e18bb0c
collect garbage at end of each slot (#975) 2020-05-06 12:01:19 +02:00