Commit Graph

1356 Commits

Author SHA1 Message Date
Zahary Karadjov 9363169098 Cosmetic improvement for the statusbar 2020-06-11 17:40:08 +03:00
Zahary Karadjov cf6a869e9e Address some TODO items; Handle start-up before genesis more properly 2020-06-11 17:40:08 +03:00
Zahary Karadjov c773e10c1a Attempt to reduce the risk of dropped network connections during the loading of KeyStores 2020-06-11 17:40:08 +03:00
Zahary Karadjov 25821331c4 More greppable code for the onPeerConnected operation 2020-06-11 17:40:08 +03:00
Zahary Karadjov 811ba9aacd Fix SSZ issues on 32-bit platforms 2020-06-11 17:40:08 +03:00
Zahary Karadjov fdaf419e41 Address review comments 2020-06-11 17:40:08 +03:00
Zahary Karadjov 6e88a07b51 Fix the validator client build 2020-06-11 17:40:08 +03:00
Zahary Karadjov 2acda1c115 Provide a default value for secretsDir (similar to validatorsDir) 2020-06-11 17:40:08 +03:00
Zahary Karadjov a75c632f7a Fixed launch_local_testnet; Renamed validator_keygen to keystore_directories 2020-06-11 17:40:08 +03:00
Zahary Karadjov a8113cf2bc Restore the local sim to a working state 2020-06-11 17:40:08 +03:00
Zahary Karadjov 17343442ea Implement more of the KeyStore spec and integrate it in the beacon node 2020-06-11 17:40:08 +03:00
Eugene Kabanov 1fc9413c48
Fix #1153. (#1160)
Add ability for SyncQueue to recover from unexpected MissingParent.
2020-06-11 16:20:53 +02:00
Jacek Sieka 016cc22173
show peer info on connect (#1155) 2020-06-11 07:14:26 +02:00
Viktor Kirilov e7febc2e2b can finalize successfully with external VCs! + some other cleanup 2020-06-10 13:50:50 +03:00
Viktor Kirilov 06873abde3 fixed build 2020-06-10 13:50:50 +03:00
Viktor Kirilov 3bae40ae91 - extracted the commands to run a VC into a separate run_validator.sh script
- each BN gets half of its previous validators as inProcess and the other half goes to the respective VC for that BN - using separate data dirs where the keys are copied
    - also removed a few command line options which are no longer necessary
- block proposals originating from a VC are propagated from one BN to the rest properly
- other cleanup & moving code back to  since it is no longer used elsewhere
2020-06-10 13:50:50 +03:00
Mamy Ratsimbazafy ce897fe83f
[Split fork choice PR] Derisk-ed attestation checks changes (#1154)
* Derisked attestation pool improvements

* tune down frequent logs

* VoteTracker logging
2020-06-10 08:58:12 +02:00
Dustin Brody a9f73ac8fb poke 2020-06-09 18:39:05 +00:00
Dustin Brody 8f7541959a allow compile-time switching between v0.11.3 and v0.12.1 2020-06-09 18:39:05 +00:00
Dustin Brody 36f15a3688 complete re-implmentation of reasonable asymptotic-complexity algorithms from 0.11.3 2020-06-09 18:39:05 +00:00
Dustin Brody 35f67cec60 remove the scaffolding get_empty_per_epoch_cache() calls and the pointless construction and return/discarding of zero-filled arrays 2020-06-09 18:39:05 +00:00
Dustin Brody 5582ecc1f4 remove some debugging code 2020-06-09 18:39:05 +00:00
Dustin Brody cc13153fc6 initial 0.12.1 beacon chain spec 2020-06-09 18:39:05 +00:00
Jacek Sieka 1e71388093
Include finalization, head and time (in that order) in standard bar (#1142)
These are the 3 most important things in the life of a beacon-node -
finalization checkpoint for security, head + time for seeing that we're
synced and working.

The order given is the expected increasing order that things should be
moving in - finalization lags head, and head lags clock - the structure
helps the user remember, but also see the pattern where time is getting
out of sync with head.

A validator would also want to know when the next action is going to be
(ie next attestation / block production) but that's for another day.
2020-06-08 18:04:11 +02:00
Eugene Kabanov 040e38529a
Fix #1140. (#1143)
SeenTable is now able to hold peers with different timeout values.
2020-06-08 18:02:50 +02:00
tersec a50415a719
temporary workaround for witti attestation dropping (#1131) 2020-06-08 09:22:31 +02:00
Eugene Kabanov 3ce98d5bca
Add checks for penalties which are not applied immediately. (#1139)
Change default maxHeadAge value to 1 epoch.
Set zero-point at the SyncQueue's initialization.
Remove annoying logs in runDiscoveryLoop.
2020-06-07 17:36:24 +02:00
Jacek Sieka ea8f96d284
ncli_db: allow saving states by root (#1136)
also dump state+block when validation fails
2020-06-06 13:26:19 +02:00
Jacek Sieka bb698a7e16
lots of peers (#1130) 2020-06-05 22:25:01 +02:00
Zahary Karadjov 4f1c4687e4
Fix the inspector build 2020-06-05 20:06:24 +03:00
Zahary Karadjov 0c78fc39e7
Use the latest LibP2P 2020-06-05 19:34:12 +03:00
Zahary Karadjov 32bc4d9096 Use {.push raises.} in spec/datatypes
This was an attempt to trigger a previously reported crash, but
everything turned out to be fine.
2020-06-05 19:33:30 +03:00
Dustin Brody 74dc2fffa6 3x blocksim speedup by using EpochRef in attestation pool addResolved(...) 2020-06-05 13:02:35 +00:00
Jacek Sieka 68b5638da4 allow non-power-of-2 limits in hashlist
fixes "make eth2_network_simulation"

a bit sad: no test coverage except against our own tests
2020-06-05 14:51:22 +03:00
Dustin Brody 3cb7896bab 12x speedup on state sim with 100k validators sans BLS by caching get_beacon_proposer_index(...) 2020-06-04 17:07:51 +00:00
Jacek Sieka bcbfa736c9
format ErrorMsg messages reasonably (#1109) 2020-06-04 08:19:25 +02:00
Jacek Sieka 56ffb696be
reorder ssz (#1099)
* reorder ssz

* split into hash_trees and ssz_serialization, roughly, for hashing and
IO
* move bitseqs into ssz (from stew)
* clean up imports

* docs, imports
2020-06-03 15:52:02 +02:00
cheatfate 405e9db199
Fix problem of good peers is also logged as timed out. 2020-06-03 13:48:01 +03:00
cheatfate 12e28a1fa9 Add proper concurrent connections.
Add SeenTable to avoid continuous attempts to dead peers.
Refactor onSecond.
Block backward sync while forward sync is working.
SyncManager now checks responses according corresponding requests + tests.
SyncManager now watching for not progressing local_head_slot and resets SyncQueue.
2020-06-03 12:53:57 +03:00
Jacek Sieka 07393c8de6
Ssz cache (#1084)
* wip: cache

* cache lists and arrays of complex objects (5x block processing speed
on ncli_db)

trivial baseline cache that stores tree in flat memory structure

* support array of uint64

* work around type issues

* more type compiler bug workarounds

* cache balances, more type fixes

* index type

* ncli_db: add validation flag, better ux

* int64 fixes

* test fix

* "oops"

```
647.913,        0.000,      647.913,      647.913,            1,
Initialize DB
0.540,        0.402,        0.340,        9.451,          619,
Load block from database
40.268,        0.000,       40.268,       40.268,            1,
Load state from database
0.498,        0.150,        0.343,        0.930,          596,
Apply block
3.548,       11.005,        0.729,       54.022,           23,
Apply epoch block
```

* support all basic types

* cleanups

* a few more cleanups
2020-06-03 07:57:17 +02:00
tersec b7bb62d48b
use shuffled validator cache in more places; cleanups (#1095) 2020-06-03 07:42:08 +02:00
Jacek Sieka 1d9a58f1e8
a few more cleanups 2020-06-02 22:10:45 +02:00
kdeme 06f025b228 Add timeout to switch.connect 2020-06-02 23:06:11 +03:00
Jacek Sieka 7e881a4c09
cleanups 2020-06-02 17:17:07 +02:00
Jacek Sieka 872d7ff493
support all basic types 2020-06-02 11:44:51 +02:00
Jacek Sieka 00acf4f7b2
"oops"
```
647.913,        0.000,      647.913,      647.913,            1,
Initialize DB
0.540,        0.402,        0.340,        9.451,          619,
Load block from database
40.268,        0.000,       40.268,       40.268,            1,
Load state from database
0.498,        0.150,        0.343,        0.930,          596,
Apply block
3.548,       11.005,        0.729,       54.022,           23,
Apply epoch block
```
2020-06-01 22:37:26 +02:00
Dustin Brody 1a6b83a15f cleanups 2020-06-01 19:45:21 +00:00
Jacek Sieka 405f460f9b
int64 fixes 2020-06-01 17:35:42 +02:00
Zahary Karadjov ef90a45509
Bump testutils 2020-06-01 16:58:28 +03:00
Jacek Sieka 2316ac15c3
index type 2020-06-01 14:52:52 +02:00
Jacek Sieka fec01829c4
cache balances, more type fixes 2020-06-01 14:52:52 +02:00
Jacek Sieka be92842944
more type compiler bug workarounds 2020-06-01 14:52:52 +02:00
Jacek Sieka a0b8b818f3
work around type issues 2020-06-01 14:52:52 +02:00
Jacek Sieka 4fc9a84c57
support array of uint64 2020-06-01 14:52:52 +02:00
Jacek Sieka f1dcee2749
wip: cache
* cache lists and arrays of complex objects (5x block processing speed
on ncli_db)

trivial baseline cache that stores tree in flat memory structure
2020-06-01 14:52:50 +02:00
tersec a327e8581b
switch state transition caching to match EpochRef (#1089)
* switch state transition caching usage to shuffled active validator indices to match EpochRef

* refactor the EpochRef -> StateCache transformation; elide pointless mapIt

* limit state passed between get_beacon_committee(...) and compute_committee(...)

* tweaks
2020-06-01 09:44:50 +02:00
Zahary Karadjov 0bcdabfcdf Detect another invalid SSZ input found through fuzzing
The first offset of an SSZ object should always have a fixed constant
value. Otherwise, some unused bytes may appear between the fixed portion
and the dynamic portion.

Please note that this fix shutds down the minimal forward compatibility
currently supported by the SSZ format (and thus, the expected behavior
must be clarified in the SSZ spec).
2020-05-30 14:47:35 +03:00
Ștefan Talpalaru a90b0dd197
Merge pull request #1077 from status-im/timeout
Eth2Node.stop(): trace msg on timeout
2020-05-29 16:30:13 +02:00
Zed c570abf602 Check iv length for decryption 2020-05-29 16:38:55 +03:00
Zed 80ce960afa Minor improvements 2020-05-29 16:38:55 +03:00
Zed 49c083d5b7 Catch all exceptions 2020-05-29 16:38:55 +03:00
Zed 14ad100b45 Address comments 2020-05-29 16:38:55 +03:00
Zed 8496e20a78 Implement EIP 2335 compliant keystore
Closes #1024
2020-05-29 16:38:55 +03:00
Kim De Mey e33c8d9067
Bump nim-eth and accompanying discv5 cleanup (#1081) 2020-05-29 12:03:29 +02:00
tersec b5f45db5e9
keep cache of per-epoch items in block pool (#1068)
* plumbing between block pool and state transition functions around active validator indices and committees

* have shared epochrefs followed by blockref tree while allowing for skipped slots

* factor out the epoch info extraction; document how the EpochRef follows forks
2020-05-29 08:10:20 +02:00
Jacek Sieka 061be037d1 ncli_db: database tool
includes a benchmark tool for now
2020-05-28 17:43:02 +00:00
Zahary Karadjov 0493737d7a Remove ssz.loadFile 2020-05-28 19:15:16 +03:00
Zahary Karadjov 1fd87a2255 Remove SszReader.maxObjectSize 2020-05-28 19:15:16 +03:00
Zahary Karadjov d4fa2c8913 Avoid an unnecessary copy in readSszValue 2020-05-28 19:15:16 +03:00
Jacek Sieka 693bc15919 avoid some RVO bugs 2020-05-28 19:15:16 +03:00
Jacek Sieka f53b55cbe0 SSZ cleanup
* be stricter about SSZ length prefix
* compute zeroHash list at compile time
* remove SSZ schema stuff
* move SSZ navigation to ncli
* cleanup a few leftover openArray uses
2020-05-28 19:15:16 +03:00
Viktor Kirilov cefd525ab3 more work on the BN/VC split
- fixed comments from the last review
- getting more data VIA RPC, moved some code back into the BN only
- attestation duties being requested as well
2020-05-28 13:01:42 +03:00
Jacek Sieka 18eca263b1
avoid stack allocations in some tests (#1079) 2020-05-28 10:28:14 +02:00
Eugene Kabanov 21131e629b
Sync freeze fixes. (#1072)
* Add ability to reset state of sync manager.
Fix bug when sync got stuck on `zero-point` reset.
Fix bug when sync got stuck when some of the workers waiting for failing one.

* Remove debugging comments and imports.

* Remove not used pendingLock.
2020-05-28 07:02:28 +02:00
Ștefan Talpalaru 273a912ae0
Eth2Node.stop(): trace msg on timeout 2020-05-28 03:14:01 +02:00
kdeme 1ec7d6e168 Quickfix for toEnode that fails 2020-05-27 13:49:18 +03:00
Zahary Karadjov 28128f4d2f Add a handler for the Goodbye message
The lack of body of `goodbye` in sync_protocol.nim was preventing
the respective LibP2P protocol to be mounted and advertised on the
network.

Adding a body fixes that, but I've also made some changes in the
P2P protocol codegen that will prevent the issue from happening
again (no body is now considered the equivalent of having an empty
body).
2020-05-26 22:17:26 +03:00
Jacek Sieka 0d54073ec9 properly work around nim type bug in ssz
the cast worked around the bug at compile time by means of casting, but
introduced a runtime error, because the pre-cast type was still being
used during deserialization
2020-05-26 14:12:42 +03:00
Dustin Brody 0929d90d93 unexport candidate_chains.init; some spec version bumps 2020-05-26 05:06:37 +00:00
Zahary Karadjov b6c9de968f
Fix some build problems and warnings 2020-05-25 20:33:03 +03:00
Viktor Kirilov 8760494c72 first batch of work towards the VC/BN split:
- we have a new binary which connects via RPC to the respective BN and has an internal clock - waking it up on every slot
- the BN has a new option called --external-validators and currently in order to have the VC binaries to run we need to pass EXTERNAL_VALIDATORS=yes to make
- factored some code out of beacon_node.nim for easier reuse in validator_api.nim and validator_client.nim
- the VC loads its associated private keys from the datadir for its BN
- most of the validator API calls have been implemented as a stub.
- the VC polls its BN at the start of each epoch - getting a list of all active validators for the current epoch - and then continues to request blocks and sign them with its appropriate validators when necessary
2020-05-25 16:23:15 +03:00
zah 626e51a91b [skip ci] Address core review comments 2020-05-24 19:00:34 +03:00
Zahary Karadjov 4a25bec2b5 Quick and dirty fix for the inspector build 2020-05-24 19:00:34 +03:00
Zahary Karadjov 833f19e942 Reform the networking layer in order to handle the new stricter SSZ API 2020-05-24 19:00:34 +03:00
Zahary Karadjov a8003e7e38 More complete integration of the List type; Detect more invalid inputs; Enable more tests 2020-05-24 19:00:34 +03:00
Zahary Karadjov accd5fe954 Don't use StackArray in ssz; Drop the support for strings 2020-05-24 19:00:34 +03:00
Dustin Brody 74e35c464f halt processing if one of the operation functions fails (failed deposits due to BLS signature issues return as not-failed, so it remains consistant with that 2020-05-24 08:20:13 +00:00
Dustin Brody 67ba11cc1a use template to more cleanly implement for_ops 2020-05-24 08:20:13 +00:00
Dustin Brody 95ee0218db switch too-strict/against-spec operations processing to use spec-based process_operations 2020-05-24 08:20:13 +00:00
Jacek Sieka f06df1cea6 remove some copies
* in makeBeaconBlock - use rollback instead
* in tests - this helps state_sim give more accurate data and makes it
30% faster
* fix some usages of raw BeaconState
2020-05-22 17:15:35 +00:00
Jacek Sieka 681b399e2a
avoid memory alloc when padding (#1058)
also avoid stack state copy in test
2020-05-22 10:31:19 +02:00
tersec 80b8416305
skip skipped constant sanity checks also from markdown reports' perspectives (#1053)
* update more spec refs in beacon_chain/spec/presets; skip skipped constant sanity checks also from markdown reports' perspectives

* mark skipped as skipped in markdown
2020-05-21 19:56:09 +02:00
Jacek Sieka 7fbb8c0bc2
return block result details (#1049) 2020-05-21 19:08:31 +02:00
Ștefan Talpalaru b2193f1b8f
Eth2Node.stop(): 5s timeout 2020-05-21 00:06:01 +02:00
Jacek Sieka a38eddcaac
remove ssz stint support (#1046) 2020-05-20 19:05:22 +02:00
Dustin Brody b9817867f5 audit & update spec references for some slashing and rewards and penalty functions; check and update all remaining v0.11.0 spec references to v0.11.3 [skip ci] 2020-05-20 14:57:08 +00:00
Jacek Sieka 23daa966be
better deserialization log 2020-05-20 15:41:02 +02:00
Jacek Sieka 30f84224a4 create dumpdir in beaconnode binary 2020-05-20 11:03:24 +00:00
Dustin Brody 3fa2f70853 update all remaining beacon chain spec references from v0.11.1 and v0.11.2 to v0.11.3 2020-05-20 08:48:18 +00:00
Jacek Sieka 418888f1d0
use ethkeys to parse enr key 2020-05-20 07:35:34 +02:00
Ștefan Talpalaru 383b22795c
bump submodules (#1043) 2020-05-20 06:57:39 +02:00
Ștefan Talpalaru c4462af4ab
beacon_node: graceful shutdown (#1033)
* beacon_node: graceful shutdown

* separate BeaconNodeStatus and BeaconNode instances
2020-05-19 20:57:35 +02:00
Dustin Brody 6c941b1ff7 spec v0.11.3 update 2020-05-19 17:16:55 +00:00
Mamy Ratsimbazafy c014f0b301
Split quarantine (#1038)
* split blockpool into hotDB and Quarantine

* Rename hotdb -> dag/candidate chains
2020-05-19 16:18:07 +02:00
Zahary Karadjov a99977b772 Progress towards #991; Enable the distinct ssz.List type 2020-05-19 15:53:03 +03:00
Eugene Kabanov ea95021073
Fix sync issues. (#1035)
* Fix sync issues.

* Add documentation about zero-point.
Add more comments about syncing loops.
Change to 4 blocks per request.
2020-05-19 14:08:50 +02:00
Jacek Sieka 4359147efc
fix snappy debug message 2020-05-19 09:23:17 +02:00
tersec 8432932c11
update to spec v0.11.2 to keep compatible with Schlesi (#1036)
* update to spec v0.11.2 to keep compatible with Schlesi

* update spec/state_transition_epoch spec references to v0.11.2

* bump other spec refs
2020-05-18 23:05:27 +00:00
Dmitriy Ryajov 0649d47df0 use proper transport flags 2020-05-18 21:51:03 +00:00
Jacek Sieka a1689f2aaf error when snappy frame is short 2020-05-18 21:15:04 +00:00
Jacek Sieka 6be7d64e8c
bump libp2p (#1031) 2020-05-18 10:11:21 +02:00
Zahary Karadjov 24a17f5814 Fix an RPC error in Lighthouse triggered by the getMetadata message 2020-05-16 09:56:13 +03:00
tersec 74db0f3c8d
fix some XDeclaredButNotUsed hints (#1027) 2020-05-15 14:41:00 +02:00
Mamy Ratsimbazafy 1916e1d2da
Revive nbench (#1022)
* Fix nbench compilation with HashedBeaconState

* Add nbench to tooling

* use newClone - fix 265e01e404 (r425198575)

* Detail advance_slot and hashTreeRoot

* Report throughput

* Fallback for ARM

* windows does not support inline ASM
2020-05-15 00:54:10 +02:00
Dustin Brody 2ddc507e5b update datatypes v0.11.1 spec references to v0.11.2 2020-05-14 16:33:11 +00:00
Dustin Brody 2d691b2e14 bump spec ref versions 2020-05-14 16:41:46 +03:00
Dustin Brody 2db99545cd allow inspector tool to log interop attestations 2020-05-14 16:41:46 +03:00
Dustin Brody 57519bebac remove some unused imports, add tests for pubsub topics, and subscribe to interop attestations 2020-05-14 16:41:46 +03:00
Jacek Sieka 3c98f18d47 document snappy decompression assumptions 2020-05-14 16:41:19 +03:00
Jacek Sieka a605c7244e simplify libp2p snappy
* handle a few more exceptions gracefully (in libp2p also)
* unify libp2p varint parsing
* decompress directly into seq
* avoid seq slice
* stop oversized snappy processing earlier (lowers risk)
2020-05-14 16:41:19 +03:00
Viktor Kirilov 926d1c546e updated news, also no longer allowing event handlers to raise anything other than a Defect in the Web3 API 2020-05-14 14:10:26 +03:00
Diederik Loerakker be806e6801
API fixes - minimal changes to HTTP RPC for Eth2Stats support (#1017)
* add api for eth2stats

* fix chain head rpc
2020-05-14 11:18:08 +02:00
Zahary Karadjov 75c1c6a95c Enable Snappy by default (using LibP2P steams for now)
This refactors the newly added Snappy streaming back-ends trying to
make them more similar and to reduce the code duplication to a minimum.
2020-05-13 12:18:42 +03:00
Zahary Karadjov f055fad08a Make the Snappy FastStreams integration optional by duplicating it for LibP2P streams 2020-05-13 12:18:42 +03:00
Zahary Karadjov 3ecb197635 Memory-optimal hash_tree_root
The work on this was started last week while I was waiting
for a decision on the "Async Snappy" PR. It was prompted by
a failing test in the test suite, where the HashingStream
was inserting some incorrectly padded chunks that affected
the result of `hash_tree_root`. Instead of working around
the problem in the HashingStream, I've decided to implement
a planned optimisation that allows us to remove the hashing
stream altogether.

With the optimisation in place, `hash_tree_root` will now
use only stack memory and only the precise amount neccesary
to build the chunks-merging tree.
2020-05-13 12:18:42 +03:00
Zahary Karadjov 7e846a0bce Implement sszSize 2020-05-13 12:18:42 +03:00
Zahary Karadjov 15f0153441 Cosmetic improvements 2020-05-13 12:18:42 +03:00
Zahary Karadjov e8a15aa0ce Document the SSZ hashing stream 2020-05-13 12:18:42 +03:00
Zahary Karadjov 9538b60704 Integrate the async Snappy implementation 2020-05-13 12:18:42 +03:00
Zahary Karadjov a739d7e8d6 Adapt SSZ to the latest FastStreams API 2020-05-13 12:18:42 +03:00
tersec 9ad05d44e4
reintroduce finalization verification to beacon node, because otherwise the consensus layer can get detached without noticing it (#1013) 2020-05-13 08:36:33 +00:00
Eugene Kabanov da0b1a4993
Fix status handling. (#1008)
* Fix status handling.
Add log map of received blocks.

* Fix review comments.
Fix UnusedImport in sync_protocol.nim
2020-05-13 08:37:58 +02:00
Dustin Brody a58d45b737
remove potentially GC leak-inducing acyclicity tag 2020-05-12 20:20:24 +02:00
tersec ba1d7e2ed4
switch state cache to use ref statedata objects to limit memory usage (#1007)
* switch state cache to use ref statedata objects to limit memory usage

* more directly initialize ref StateData

* use HashedBeaconState instead of StateData to try to fix memory leak

* switch cache to seq[ref HashedBeaconState]

* remove unused import

Co-authored-by: Ștefan Talpalaru <stefantalpalaru@yahoo.com>
2020-05-12 16:26:58 +00:00
Jacek Sieka fb2e0ddbec
sync fixes (#1005)
* sync fixes

* fix Status message finalized info
* work around sync starting before initial status exchange
* don't fail block on deposit signature check failure (fixes #989)
* print ForkDigest and Version nicely
* dump incoming blocks
* fix crash when libp2p peer connection is closed
* update chunk size to 16 to work around missing blocks when syncing

* bump libp2p

* bump libp2p

* better deposit skip message
2020-05-11 18:08:52 +00:00
Ștefan Talpalaru a7a50824a1
more metrics (#1004) 2020-05-11 06:25:49 +00:00
tersec c498103b2f
quick/minimal mitigation of beacon_node memory usage resulting from 2*Table.defaultInitialSize pointless BeaconState objects in block pool state cache (#1002) 2020-05-10 16:31:55 +00:00
Jacek Sieka 592d6f65c3
bump (#1001) 2020-05-09 16:47:14 +02:00
Eugene Kabanov 55dfcc6783
Inspector native version. (#995) 2020-05-09 14:18:58 +00:00
tersec 093d298b2b
Increase finalization and finalization checking robustness (#990)
* fix some warnings related to beacon_node splitting; reimplement finalization verification more robustly; improve attestation pool block selection logic

* re-add missing import

* whitelist allowed state transition flags and make rollback/restore naming more consistent

* restore usage of update flags passed into skipAndUpdateState(...) in addition to the potential verifyFinalization flag

* switch rest of rollback -> restore
2020-05-09 12:43:15 +00:00
Ștefan Talpalaru c572f61129
bump vendor/nim-metrics 2020-05-09 01:13:57 +02:00
Ștefan Talpalaru fc9d6bc1b3
enable SO_REUSEADDR in libp2p transports 2020-05-08 21:50:20 +02:00
Jacek Sieka c1aff83b8b
decrease sync chunk size (#988)
While mainnet performance is being worked on, this helps with the flow.
2020-05-08 08:17:40 +02:00
Eugene Kabanov 3ffd0d7216
Fix gossip messages not delivered. (#983) 2020-05-06 15:42:59 +00:00
Mamy Ratsimbazafy 621c2e38a6
Separate validator duties from the Beacon Node (#976)
* Separate validator duties from the Beacon Node

* remove duplicate MaxEmptySlotCOunt

* imports

* have beacnde properly compile
2020-05-06 13:23:45 +00:00
Jacek Sieka a56e18bb0c
collect garbage at end of each slot (#975) 2020-05-06 12:01:19 +02:00
tersec c548b90a0e
default preset update; rm process_slots(BeaconState); some BLS skipping disabled (#963)
* re-enable BLS validation for block sanity tests

* update default preset to mainnet; remove process_slots(state: var BeaconState)

* bump spec ref to v0.11.1

* bump all v0.11.1 spec references except the actually changed is_valid_indexed_attestation(...) in spec/beaconstate to v0.11.2

* remove stack-risky/low-ROI doAssert
2020-05-04 21:07:18 +00:00
Jacek Sieka 01e9df97cb
cleanups (#962)
* remove broken serialized_sizes
* actually use sszdump module
* avoid bitops
* fix stack_sizes module name
2020-05-04 07:38:14 +02:00
Jacek Sieka 2449d4b479
cache empty slot state root (#961)
When replaying state transitions, for the slots that have a block, the
state root is taken from the block. For slots that lack a block, it's
currently calculated using hash_tree_root which is expensive.

Caching the empty slot state roots helps us avoid recalculating this
hash, meaning that for replay, hashes are never calculated. This turns
blocks into fairly lightweight "state-diffs"!

* avoid re-saving state when replaying blocks
* advance empty slots slot-by-slot and save root
* fix sim randomness
* fix sim genesis filename
* introduce `isEpoch` to check if a slot is an epoch slot
2020-05-03 19:44:04 +02:00
Jacek Sieka a3e098cf92
block pool simulator (#956)
* block pool simulator

like state_sim, but more
2020-05-01 17:51:24 +02:00
tersec cf8e90615a
More state_transition unification (#953)
* remove incorrect/obsolete comment; deprecate BeaconState state transition functions

* remove deprecated state_transition(state: var BeaconState)

* add specific workarounds for state_transition() and process_slots() to nfuzz_block() and addTestBlock()
2020-04-30 18:27:17 +02:00
Jacek Sieka e65f5c86da
vendor: bump (#954) 2020-04-30 18:27:04 +02:00
tersec cd939d4606
fix spec/beaconstate compile error in the presence of other PRs' SSZ tightening (#952) 2020-04-30 09:01:55 +00:00
tersec 3a56ddc5c4
begin removing plain BeaconState versions of state transition functions (#951)
* remove near-duplicate code paths: process_slot(), process_slots(), and state_transition() for BeaconState are now wrappers around the HashedBeaconState versions

* convert tests/test_state_transition.nim to use HashedBeaconState

* convert mocking infrastructure and spec_block/epoch_processing tests to use HashedBeaconBlock, and remove thus unused process_slot*(state: var BeaconState)
2020-04-30 08:44:19 +02:00
Jacek Sieka c74ba5c0c6
ssz: move ref support outside (#943)
* ssz: move ref support outside

Instead of allocating ref's inside SSZ, move it to separate helper:

* makes `ref` allocations explicit
* less magic inside SSZ
* `ref` in nim generally means reference whereas SSZ was loading as
value - if a type indeed used references it would get copies instead of
references to a single value on roundtrip which is unexpected

TODO: EF tests would benefit from some refactoring since they all do the
same thing practically..

Co-authored-by: Zahary Karadjov <zahary@gmail.com>
2020-04-29 22:12:07 +02:00
cheatfate 49cc9a9961 Fix updateStatus should not raise exceptions. 2020-04-29 19:21:18 +03:00
tersec 57aba5d3a6
Switch block pool caches from BeaconChainDB to TableRefs (#945)
* refactor blook pool caches to directly use TableRef to avoid SSZ decoding, which was consuming 20% of profile on mainnet eth2_network_simulation

* use table's hasKeyOrPut

* bump eth2 spec reference to v0.11.1

* cache whole StateData objects and switch from expensive clear() to cheaper new object instantiation for caching

* remove scaffolding and stop re-assigning to part of StateData object

* 80-character lines
2020-04-29 16:58:44 +02:00
Zahary Karadjov 667bdf066a Bump Chronicles to fix a minor bug; Add a placeholder for disabled topics 2020-04-28 13:49:30 +03:00
Zahary Karadjov 7237a370ba Implement https://github.com/status-im/nim-beacon-chain/issues/872
Please see the newly added 'schlesi-dev' Makefile target.
It demonstrates how the log level can be specified for individual topics.

Additionally, when connecting to testnets like 'schlesi' there will be
two additional log files produced in the working directory:

* json-log.txt
* text-log.txt (in the textblocks format)
2020-04-28 13:49:30 +03:00
Jacek Sieka 28d6cd2524
avoid memory allocations and copies when loading states (#942)
* rolls back some of the ref changes
* adds utility to calculate stack sizes
* works around bugs in nim exception handling and rvo
2020-04-28 10:08:32 +02:00
Zahary Karadjov 80b538452e Updated the custom preset loader to the latest spec; Working makefile target for connecting to the Schlesi testnet 2020-04-27 23:22:54 +03:00
Viktor Kirilov dcc70c5259 resolved final comment from review 2020-04-27 22:10:35 +03:00
Viktor Kirilov 8ef28c905d fixed commets from review 2020-04-27 22:10:35 +03:00
Viktor Kirilov 5ecba6df49 broadcasting and listening to the '_snappy' topics as well 2020-04-27 22:10:35 +03:00
Jacek Sieka 898df9ba45
kvstore: port to nim-eth (#938) 2020-04-27 18:36:28 +02:00
tersec 7790644e52
remove a pointless hash_tree_root(BeaconState) per node per proposed block (#933)
* remove a pointless hash_tree_root(BeaconBlock)

* use ref with putState
2020-04-27 12:47:49 +02:00
Jacek Sieka 03a147ab8d
avoid state copy in state transition (#930)
In BlockPool, we keep the head state around, so it's trivial to restore
the temporary state there and keep going as if nothing happened.

This solves 3 problems:
* stack space - the state copy on mainnet is huge
* GC scanning - using stack space for state slows down the GC
significantly
* reckless copying - the copy itself takes a long time

In state_sim, we'll do the same and allocate on heap - this helps a
little with GC - without it, the collection of the temporary strings
created with `toHex` while printing the json dominates the trace.
2020-04-26 21:13:33 +02:00
tersec be475a82d7
catch invalid 0-raw-byte Bitlist SSZ inputs during parsing (#936)
* catch invalid 0-raw-byte Bitlist SSZ inputs during parsing

* bump a couple more spec refs to v0.11.1
2020-04-26 16:26:53 +00:00
Zahary Karadjov 4e9fa51ae9 Introduce BeaconNodeRef and use it in all the right places 2020-04-26 13:04:53 +03:00
Zahary Karadjov fdcbfdff05 Pass the test suite with a BeaconState ref type 2020-04-26 13:04:53 +03:00
Zahary Karadjov 740b76d152 Spec-compliant implementation of Eth1 monitoring; Eth1-enabled local sim
BEWARE! This commit will trigger a stack overflow during local sim
2020-04-26 13:04:53 +03:00
tersec 65353bab76
add another check for inconsistent aggregation and committee length (#927)
* add another check for inconsistent aggregation and committee length, since ncli_transition bypasses process_attestation(...)/check_attestation(...) and calls almost directly into process_epoch(...)

* bump validator functions to v0.11.1 spec references

* bump some spec references to v0.11.1

* poke
2020-04-24 09:16:40 +02:00
Jacek Sieka 494ffb63ce
eh fixes (#926)
* work around improbable exceptions in metrics / chronos
* fix unnecessary lookup in block pool
2020-04-24 09:16:11 +02:00
tersec 7bc18423fa
detect improper 0-length lists of variable-sized objects in SSZ reading (#928)
* detect improper 0-length lists of variable-sized objects in SSZ reading
2020-04-23 19:39:23 +00:00
Viktor Kirilov 0079242457 again exported the right overloads for serializing hashes properly to json (and not as byte arrays) and updating nim-eth 2020-04-23 20:46:13 +03:00
Viktor Kirilov 4c66999a43 removing the json serialization change - fails to compile with -d:testnet_servers_image and LOG_LEVEL=TRACE and isn't worth the time at the moment 2020-04-23 20:46:13 +03:00
Viktor Kirilov 1575ded589 exported the right overloads for serializing hashes properly to json (and not as byte arrays) 2020-04-23 20:46:13 +03:00
Viktor Kirilov f3d0ae35f6 added the attestation bitfield to the bootstrap enr 2020-04-23 20:46:13 +03:00
Eugene Kabanov be89a3c54d
Add "drop by score" ability to PeerPool. (#917)
* Add "drop by score" ability to PeerPool.
Add tests.
Fix syncmanager queue to start from most fresh data.

* Fix endless cycle at the end of syncing process.
2020-04-23 17:31:00 +02:00
tersec 7cd4b0bfae
reduce stack space usage in process_final_updates(...) (#925)
* reduce stack space usage in process_final_updates(...) to avoid fuzzed segfault in https://github.com/status-im/nim-beacon-chain/issues/921

* document motivation behind manually constructing hash_tree_root of a HistoricalBatch
2020-04-23 13:39:38 +02:00
tersec 7e94482409
avoid rewindStateData/state_transition in fast-path (#919)
* avoid rewindStateData/state_transition totally in fast-pathed updateStateData calls

* document motivation behind BlockPool.getStateDataCached(...) and clarify naming
2020-04-23 11:59:29 +02:00
Jacek Sieka 6729d3c032
kvstore: fix raising, be harsher on database errors (#923)
* kvstore: fix raising, be harsher on database errors

* bump stew/serialization
2020-04-23 08:27:35 +02:00
Jacek Sieka 65ca74c980 req: cap requested blocks better
also cap blocks in roots request
2020-04-22 12:09:26 +03:00
Jacek Sieka ed74770451
spec: regulate exceptions (#913)
* spec: regulate exceptions

* a few more simple raises
2020-04-22 07:53:02 +02:00
tersec bdea75e4c3
Restore all blockpool extended validation checks (#886)
* fix remaining block pool extended validation issues and re-enable first-block-received and block-signature EV checks; enable Merkle validation in beacon_node in eth2_network_simulation; refactor some Merkle proof generation code outside tests/ as a result

* re-enable Merkle validation skipping, since while it works on make eth2_network_simulation, it has issues with local testnet

* tighten already-seen-block blockpool check; move comment closer to conceptually proximate code; queue up maybe-future-valid-blocks as pending to keep libp2p-synchronous interrupt handling time lower

* revert the cleanups, now in a separate PR

* remove the remaining merkle_minimal cleanup remnants, also moved to other PR

* restore PR to only modifying one file after rebasing

* use signatures as summary to compare block contents

* switch signature comparison to be raw byte-wise to ensure no attempts to deserialize it to valid (or not) BLS signatures first
2020-04-21 18:52:53 +02:00
Jacek Sieka da988e4e40 more block range request updates
* handle skip == 0 gracefully
* avoid memory allocation at expense of more complex API
* add more tests
* log block request results
2020-04-21 11:07:27 +03:00
Jacek Sieka fccce85f0d simplify block request
this matches the intended spec behaviour
2020-04-21 11:07:27 +03:00
Jacek Sieka 2c2de2b35e
pull out validator duties to simplift onSlot handler (#910) 2020-04-20 19:28:12 +02:00
tersec 8a72ae89b9
fix mainnet finalization (#906)
* fix mainnet finalization and swith eth2_network_simulation to a kind of small-mainnet profile

* Fix slot reference in trace logging

* bump a couple of spec refs from v0.11.0 to v0.11.1

* bump another spec ref to v0.11.1, one more try at Jenkins test vector download CI issue

* fix other slot reference in trace logging and skip past single-block/multi-slot gaps to re-approach from ancestry side by state_transitioning, by requiring exact match on both root hash and slot for fast path

* make more precise the fast path condition

* redo logic to make uniform with BeaconChainDB; fix chronos deprecation warning

* revert not-working replacement of deprecated chronos futures `or`

* switch testnet1 to mainnet
2020-04-20 19:27:52 +02:00
Eugene Kabanov 3d42da90a8
Syncing. (#909) 2020-04-20 16:59:18 +02:00
Jacek Sieka edecce1751
disable aggregation when no validators are connected (#908)
* if we don't have validators, don't consider aggregation work
* if we do have validators, don't aggregate when we're out of sync
* when we do aggregate, use a fresh state, and not one from before
sleeping
2020-04-20 14:20:53 +02:00
tersec 480c11f8aa
fix remaining block sanity test vectors (#902)
* fix remaining block sanity test vectors

* bump epoch processing spec refs to v0.11.1
2020-04-18 00:09:34 +02:00
Dustin Brody 164d171bd9
revert previous commit 2020-04-17 20:27:42 +02:00
Dustin Brody ef92529feb
fix remaining block sanity test vectors 2020-04-17 20:25:18 +02:00
Mamy Ratsimbazafy 17a0a965fc
ambiguous calls between toHex from byteutils and nimcrypto (#901)
* Fix potential ambiguous calls between toHex from byteutils and nimcrypto due to mixin

* remove the mixins
2020-04-17 20:10:39 +02:00
Dustin Brody 3df9a22ff7 fix remaining attester slashing tests from EF phase0 spec tests by checking for uniqueness 2020-04-17 16:01:32 +00:00
Dustin Brody 05d1d1246a bump a couple more eth2 phase0 spec refs to v0.11.1 2020-04-17 16:37:52 +03:00
Dustin Brody 943a7da610 bump some eth2 phase0 spec refs to v0.11.1 2020-04-17 16:37:52 +03:00
Dustin Brody 4f970a067a fix variable offset bounding/comparison 2020-04-17 16:37:52 +03:00
Dustin Brody 71e802c740 harden SSZ reader 2020-04-17 16:37:52 +03:00
Jacek Sieka 8b6c51e78d
secp: convenience api (#897) 2020-04-17 15:29:49 +02:00
Zahary Karadjov a4830976ab
hotfix: fix the selecting of the syncing protocol handlers 2020-04-16 15:01:19 +03:00
Ștefan Talpalaru 7b8d6ea61c
bump vendor/nimbus-build-system (#894)
* bump vendor/nimbus-build-system

* change beacon_node banner

* beacon_node: update copyright years

* version.nim is imported in beacon_chain.nimble
2020-04-16 00:20:27 +02:00
Zahary Karadjov c44d66c697 Fix more typos 2020-04-16 01:01:18 +03:00
Zahary Karadjov 99f9bc4eee Fix the inspector build 2020-04-16 01:01:18 +03:00
Zahary Karadjov 672f690bf6 Implement most of the v0.11.1 networking changes 2020-04-16 01:01:18 +03:00
Zahary Karadjov 8d639d9bd7 Avoid accidental memory allocations in SSZ 2020-04-16 01:01:18 +03:00
Zahary Karadjov 22e5f8165e Don't use `defer` in eth2_network to avoid https://github.com/nim-lang/Nim/issues/13899 2020-04-16 01:01:18 +03:00
Zahary Karadjov f34fd46e3a Add snappy and protobuf-serialization; Use the latest FastStreams API 2020-04-16 01:01:18 +03:00
Dustin Brody d559f4ee89 create distinct CommitteeIndex type 2020-04-15 09:59:49 +00:00
Dustin Brody f49dbf68e6 refactor some Merkle proof generation code outside tests; remove unused variables/functions 2020-04-15 09:59:49 +00:00
Jacek Sieka 8eafa6e094
fork: push defect (#888) 2020-04-15 11:21:22 +02:00
Ștefan Talpalaru b7a32a17ba
bump submodules
and remove failing syncManagerGroupRecoveryTest
2020-04-14 18:21:56 +02:00
Viktor Kirilov cab68c28ef fixed name in logging - was causing confusing (and wrong) output regarding the filalized slots 2020-04-13 18:17:24 +03:00
tersec d25d674502
Remove more warnings, both deprecations and unused imports (#884)
* fix warnings by switching from deprecated chronos API addTimer(...) to setTimer(...) and removing especially some unnecessary chronicles and extras imports from test suite modules

* update a couple v0.10.1 spec references to v0.11.1
2020-04-11 19:41:50 +02:00
Jacek Sieka afa08c8e3c
crypto: cleanup (#882)
* crypto: cleanup

* fix several Defect-on-user-input
* make crypto interface more similar to secp
* use `crypto.nim` in all of nbc

* digest: raises

* fix

* vendor
2020-04-11 10:51:07 +02:00
tersec 31bf8fa408
remove all but one UnusedImport warning (#881)
* remove all but one UnusedImport warning

* bump a few more spec version references from v0.10.1 to v0.11.1

* more v0.10.1 spec reference updates/removals

* yet more v0.10.1 spec reference updates
2020-04-10 16:06:24 +02:00
tersec ccbbce79a9
Mostly remove skipMerkleValidation by fixing Merkle proof construction/usage (#879)
* refactor and fix merkle proof construction in test suite and thereby remove most remaining skipMerkleValidation flags, now unnecessary

* a few non-semantic comment update/removals
2020-04-10 15:59:17 +02:00
Mamy Ratsimbazafy cbc998ed93
[Ready 1/2] Fork choice rewrite (#865)
* initial fork-choice refactor

* Add fork_choice test for "no votes"

* Initial test with voting: fix handling of unknown validators and parent blocks

* Fix tiebreak of votes

* Cleanup debugging traces

* Complexify the vote test

* fakeHash use the bigEndian repr of number + fix tiebreak for good

* Stash changes: found critical bug in nimcrypto `==` and var openarray

* Passing fork choice tests with varying votes

* Add FFG fork choice scenario + fork choice to the test suite

* Not sure why lmdb / rocksdb reappeared in rebase

* Add sanity checks to .nimble file + integrate fork choice tests to the test DB and test timing

* Cleanup debugging echos

* nimcrypto fix https://github.com/status-im/nim-beacon-chain/pull/864 as been merged, remove TODO comment

* Turn fork choice exception-free

* Cleanup "result" to ensure early return is properly used

* Add a comment on private/public error code vs Result

* result -> results following https://github.com/status-im/nim-beacon-chain/pull/866

* Address comments:
- raises: [Defect] doesn't work -> TODO
- process_attestation cannot fail
- try/except as expression pending Nim v1.2.0
- cleanup TODOs

* re-enable all sanity checks

* tag no raise for process_attestation

* use raises defect everywhere in fork choice and fix process_attestation test
2020-04-09 18:15:00 +02:00
Jacek Sieka 0f47c76b50
reenable block bls verification (#875)
* finalizing state_transition

* cleanup

Co-authored-by: Dustin Brody <tersec@users.noreply.github.com>
2020-04-09 09:41:02 +00:00
Ștefan Talpalaru 96431bfd86
post-merge fixes 2020-04-08 23:09:52 +02:00
Ștefan Talpalaru b574799088
Merge branch 'devel' 2020-04-08 22:54:18 +02:00
Zahary Karadjov 2930b65eb4
[skip ci] simple logging fix 2020-04-07 19:04:49 +03:00
Jacek Sieka b4fc641b8c result->results, bump bearssl 2020-04-07 12:08:59 +00:00
Dustin Brody 2771deadfc re-add test_interop to all_tests and mark several v0.10.1 phase 0 spec references as v0.11.1 2020-04-07 13:16:55 +03:00
Dustin Brody e7de8aae20 fix/restore 0.11.1 BeaconState genesis interop: (a) use GENESIS_FORK_VERSION properly in BeaconState init; (b) use GENESIS_FORK_VERSION to be compatible with zcli/zrnt initial eth1 deposits; (c) let List[foo] work with 64-bit ints, so BeaconState.validators SSZ-serializes properly; and (d) update test_interop to use new spec/crypto API 2020-04-07 13:16:55 +03:00
Jacek Sieka e1d18b431b
reraise defect in generic exception handler 2020-04-06 20:08:39 +02:00
Jacek Sieka b9cc91aef8 keep up with nim-eth 2020-04-06 14:13:20 +03:00
Jacek Sieka 04cf6d30eb
vendor: bump nim-eth + fallout (#861) 2020-04-05 11:50:31 +02:00
Dustin Brody 60df05a420 mark several v0.10.1 spec references as v0.11.1-compatible 2020-04-04 13:43:04 +00:00
Dustin Brody f9e45dc121 document and temporary workaround for extended validation issue 2020-04-04 13:43:04 +00:00
Jacek Sieka d180724d41
Bump eth (#857)
* vendor: bump stew
* vendor: bump nim-eth
* vendor: bump web3 too
2020-04-03 11:12:38 +02:00
kdeme fc2fff249b Discv5: Don't pass ip address when external ip is not known 2020-04-02 00:52:34 +03:00
tersec ea59f2ecf2
comments-only changes: update a bunch of v0.10.1 spec references to v0.11.0 and explain rationale for extended validation design in code (#847) 2020-04-01 11:41:39 +00:00
tersec 6eb4f1f39d
initial attestation aggregation (#769)
* initial attestation aggregation

* fix usage of committee index, vs index in committee; uniformly set trailing/following distance; document how the only-broadcast-if mechanism works better and what aggregation already happens, not otherwise sufficiently clear; use correct BlockSlot across epoch boundaries

* address inconsistent notion of which slot in past to target for aggregate broadcast; follow 0.11.x aggregate broadcast p2p interface topic

* Fix get_slot_signature(...) call after get_domain(...) change required genesis_validators_root

* mark all spec references which aren't dealt with in other PRs as v0.11.1

* update two more spec refs to v0.11.1
2020-04-01 09:59:55 +00:00
tersec cd388bc9bb
extended validation (#812)
* initial extended validation setup

* flesh out all TODO items for attestation and beaconblock verification

* fix finalization and add chronicles debugging messages

* directly use blockPool.headState rather than pointlessly updating it and document this constraint

* fix logic relating to first-attestation checking; support validating blocks across multiple forks
2020-03-31 18:39:02 +00:00
tersec daabb1b5b2
complete all (non-semantic, comment-only) 0.11.0 -> 0.11.1 beacon chain spec updates, mark all beacon chain v0.11.0 spec references as v0.11.1, and remove now unnecessary/unused UpdateFlags vars from 4 test fixtures (#841) 2020-03-30 23:40:24 +00:00
Ștefan Talpalaru e37c12b810
Merge branch 'peerinfo' into devel 2020-03-30 19:55:24 +02:00
tersec f5f939bd31
0.11.1 beacon chain spec update (#836)
* initial 0.11.1 spec commit; no test regressions and finalizes in eth2_network_simulation

* with BLS 0.10/0.11 available, stop skipping attester slashing, proposer slashing, and voluntary exist operations fixture tests

* switch param orders to group state.{fork, genesis_validators_root}; bump spec/datatypes spec version for network purposes

* mark attestation construction and broadcast and some minimal/mainnet constants as 0.11.1-compatible; remove phase 1 sharding constants from minimal which don't exist in that preset
2020-03-30 11:31:44 +00:00
Ștefan Talpalaru 97abe77398
close unused PeerInfo instance
Should not be needed from the GC's point of view, but needed when
gathering Future.state statistics, due to `PeerInfo.lifefut`.
2020-03-30 00:23:01 +02:00
Zahary Karadjov 46e395c647 Switch to GossipSub 2020-03-25 23:32:05 +02:00
Zahary Karadjov 22876da593 Fix gcsafety issues in the test suite 2020-03-24 22:14:40 +02:00
Zahary Karadjov 86a3bea1ec Use the latest Chronos and Libp2p 2020-03-24 22:14:40 +02:00
kdeme b80a5b90df Adjustments to store bootnode enrs in discovery object + bump nim-eth 2020-03-24 22:14:40 +02:00
Zahary Karadjov f2434139e9
Hotfix: Don't report failure to connect to bootstrap node due to the initial sync taking more than 30 seconds 2020-03-23 22:29:50 +02:00
cheatfate d63e4870e6 Fix Stream EOF problem. 2020-03-23 22:21:27 +02:00
zah bd5400aea4
Oops, change back the default network_type in the master branch 2020-03-23 18:19:31 +02:00
Zahary Karadjov 64fbad1efa Some build fixes required for pushing testnet0 2020-03-23 16:13:11 +02:00
Zahary Karadjov 46a3009c63 Restore the address file feature (and use it for synchronization in local sim) 2020-03-23 14:59:33 +02:00
Zahary Karadjov 4623aa81ec Clean up some obsolete code 2020-03-23 14:59:33 +02:00
Zahary Karadjov d5e4e640b4 Merge libp2p_backend into eth2_network 2020-03-23 14:59:33 +02:00
Zahary Karadjov 0c018cb68a Mechanically remove all mentions of the daemon from the code 2020-03-23 14:59:33 +02:00
Zahary Karadjov 8300cee131 Merge the contents of backends_common in libp2p_backend 2020-03-23 14:59:33 +02:00
Zahary Karadjov 5868afe341
Restore compilation with -d:p2pdump 2020-03-22 19:28:11 +02:00
Zahary Karadjov 286d6e14da Attempt to fix the Travis ARM build 2020-03-22 19:00:09 +02:00
Zahary Karadjov 1d78a43724 Don't return Option[BlockRef] because BlockRef is a ref type 2020-03-22 19:00:09 +02:00
Zahary Karadjov d2b7ee27de The RPC service listens on localhost by default 2020-03-22 19:00:09 +02:00
Zahary Karadjov 7d2381240f Bump nim-eth and nim-json-rpc 2020-03-22 19:00:09 +02:00
Zahary Karadjov 19847bbd20 Remove some warnings 2020-03-22 19:00:09 +02:00
Zahary Karadjov 5a2a52869e Add a RPC proc similar to /spec/eth2_config 2020-03-22 19:00:09 +02:00
Zahary Karadjov adcec61081 Initial implementation of a JSON-RPC service 2020-03-22 19:00:09 +02:00
tersec 53e4b0a26c
complete (except for get_domain(...)) 0.11.0 beacon chain spec update (#822)
* complete (except for get_domain(...)) 0.11.0 beacon chain spec update

* mark compute_start_slot_at_epoch(...), is_active_validator(...), compute_signing_root(...), and get_seed(...) as 0.11.0
2020-03-22 16:03:07 +00:00
Jacek Sieka 689bcf71c4 clean up block creation
* consistently use state at new block slot to produce block
* factor out signature funcs
* fix missing block application test
2020-03-22 11:15:07 +00:00
Mamy Ratsimbazafy ccace9a034
Fix shortlog crypto (#818)
* fix shortlogs of crypto types

* Valid signature without a "real: " prefix tag

* remove 0x prefix
2020-03-19 17:18:48 +01:00
Dustin Brody f7ecd09e95 mark aggregate(...), AttesterSlashing, IndexedAttestation, AttestationData, Deposit, DepositData, VoluntaryExit, BeaconBlock, BeaconBlockHeader, BeaconState, HistoricalBatch, Fork, SigningRoot, SignedVoluntaryExit, SignedBeaconBlock, SignedBeaconBlockHeader, get_current_epoch(...), and get_randao_mix(...) as 0.11.0 2020-03-17 12:06:56 +02:00
Dustin Brody cae1c0fd1a address beacon_node TODO regarding missing proposer index; mark initiate_validator_exit(...), is_eligible_for_activation_queue(...), Attestation, DepositMessage, BeaconBlockBody, PendingAttestation, Eth1Data, and integer_squareroot(...) as 0.11.0 2020-03-17 12:06:56 +02:00
Dustin Brody ebcde664e5 mark decrease_balance(...), get_validator_churn_limit(...), network spec version, Checkpoint, compute_epoch_at_slot(...), get_active_validator_indices(...), and get_committee_count_at_slot(...) as 0.11.0 2020-03-17 12:06:56 +02:00
Dustin Brody 313109c743 align process_block(...) name with spec; fix beacon chain proposer index generation when some validators aren't active to fix the two regressing 0.11.0 block sanity tests 2020-03-17 12:06:56 +02:00
Dustin Brody 11300fc290 fix finalization 2020-03-17 12:06:56 +02:00
Dustin Brody 6068dd960b initial 0.11.0 spec version update 2020-03-17 12:06:56 +02:00
tersec 33687c3e41
remove lmdb (#809) 2020-03-14 11:33:37 +01:00
Mamy André-Ratsimbazafy 2aae54a15f Fix serialization of deposits 2020-03-13 12:20:41 +00:00
Mamy André-Ratsimbazafy c1c7435808 Skip tests:
- EF - attester slashings - 2 tests
- EF - sanity blocks - 1 test
- NBC - attestation - 1 test
2020-03-13 12:20:41 +00:00
Mamy André-Ratsimbazafy 88002ddb02 Fix ambiguous call newKeyPair 2020-03-13 12:20:41 +00:00
Mamy André-Ratsimbazafy 28dc8a6a29 Fix mock signatures/validator keys and stack smashing 2020-03-13 12:20:41 +00:00
Mamy André-Ratsimbazafy 65d668c3f6 Update validator keygen 2020-03-13 12:20:41 +00:00
Mamy André-Ratsimbazafy 14bc9e60ca "init" should handle fake/invalid pubkeys and signatures 2020-03-13 12:20:41 +00:00
Mamy André-Ratsimbazafy 3c140d0c3d Try to solve the stack smashing / discriminant changes object branch 2020-03-13 12:20:41 +00:00
Mamy André-Ratsimbazafy 31bd13a5d3 init for mainchain_monitor 2020-03-13 12:20:41 +00:00
Mamy André-Ratsimbazafy bfd4df912e Update to BLS v0.10.1 2020-03-13 12:20:41 +00:00
Ștefan Talpalaru ab0879958a don't log message content in P2PStream.readMsgBytes()
because it adds too much noise to the trace logs
2020-03-12 13:50:21 +02:00
cheatfate d802fa593c Add pubsub message logging.
Initial commit of logtrace tool.
2020-03-11 17:26:30 +00:00
Dustin Brody 0d3de00714 remove unused imports 2020-03-11 10:50:55 +00:00
tersec a327c82820 Revert "finalizing state_transition (#792)"
This reverts commit 91d75bdfdd.
2020-03-10 16:44:42 +00:00
Joao Gabriel Carvalho 91d75bdfdd
finalizing state_transition (#792) 2020-03-10 12:19:00 +01:00
Jacek Sieka 7902d070cd prefer endians2 2020-03-05 18:13:28 +02:00
Jacek Sieka 9a3db7a81f
[WIP] Fake bls at runtime (#735)
* Initial implementation of runtime bls skipping.

Add libnfuzz skipBLSValidation handling, check that it propagates.

* Rename skipBLSValidation -> skipBlsValidation, start skipStateRootValidation
* Replace skipValidation flags with more granular flags.

Also added skipBlockParentRootValidation flag
Mainly replaced with skipBlsValidation but also StateRoot or
BlockParentRootValidation flags where appropriate.

* Adjust interop test to pass when skipping merkle validation.
* Stop skipping validation for mainchain_monitor.
* Remove comment.
* Also skipMerkleValidation for test_beacon_chain_db.
2020-03-05 13:52:10 +01:00
kdeme c382c6434a Use peerPool len + fix removal of peerPool on disconnect 2020-03-05 13:17:34 +02:00
kdeme 02717a6354 Use randomNodes from discv5 instead of lookup 2020-03-05 13:17:34 +02:00
Joao Gabriel Carvalho 91f87b55b4
SignedBeaconBlock in state_transition (#773)
* using SignedBeaconBlock in state_transition
2020-02-29 16:15:44 +01:00
cheatfate ab1a08084b One more attempt to fix compilation error. 2020-02-28 14:44:41 +02:00
cheatfate 278b91d6ce Supporting 2 backends is hell. 2020-02-28 14:44:41 +02:00
cheatfate 5ae0026f72 Fix one more compilation error. 2020-02-28 14:44:41 +02:00
cheatfate dd316e04d1 Fix one more compilation error. 2020-02-28 14:44:41 +02:00
cheatfate 0f0a839444 Fix compilation errors. 2020-02-28 14:44:41 +02:00
cheatfate 547c8a44d0 Replace Table[PeerID, Peer] with PeerPool.
Add `score` Peer.
2020-02-28 14:44:41 +02:00
Ștefan Talpalaru c2c570fcb0
bump submodules
plus a few small modifications
2020-02-26 01:14:20 +01:00
Dustin Brody 3d99c7b28e mark some spec functions .used. to avoid warnings; misc cleanups 2020-02-25 09:17:06 +00:00
Zahary Karadjov fa0f3b4468
Use the specified extip address in the advertised ENR records 2020-02-22 20:54:15 +02:00
Dustin Brody 425b0e13da remove potential error condition in getAttestationTopic; subscribe to all topics in parallel; add test cases for larger-than-ATTESTATION_SUBNET_COUNT committee indexes 2020-02-21 19:44:03 +00:00
Dustin Brody 62584b633e update relevant portions of attestation aggregation spec version; tweak inspector 2020-02-21 19:44:03 +00:00
Dustin Brody a8567cafb5 mark out where/how AttestationData and Attestation construction are honest validator-compliant 2020-02-21 19:44:03 +00:00
Dustin Brody 32872c7ee6 bump spec version to 0.10.1 (more accurate than 0.10.0); fix inspector building 2020-02-21 19:44:03 +00:00
Dustin Brody 6c87af9aab split attestation beacon topic per honest validator spec 2020-02-21 19:44:03 +00:00
Dustin Brody bced5ca656 implement is_proposer(...) 2020-02-21 19:44:03 +00:00
Dustin Brody 4782ae06e6 implement get_committee_assignment(...) by spec transcription 2020-02-21 19:44:03 +00:00
Dmitriy Ryajov 08915ed597 bump libp2p and increase bootstrap timeout 2020-02-21 12:03:19 -06:00
Jacek Sieka c25f8d4adf
sqlite kvstore backend (#749) 2020-02-20 11:49:34 +01:00
Zahary Karadjov 9e58e22761 [skip ci] remove an obsolete comment 2020-02-20 10:58:38 +02:00
cheatfate 5742299c94 Fix freezes in connection initialization. 2020-02-20 10:58:38 +02:00
Zahary Karadjov 880b981a0a
Don't wait for 10 seconds when connecting to the network 2020-02-19 12:45:34 +02:00
Zahary Karadjov 55886ff0ac
More proper fix replacing hotfix 052a6ac3bc
nim-eth was bumped to take advantage of a fixed `==` for public eth keys
2020-02-19 11:59:38 +02:00
Zahary Karadjov 3b9e9fefa8
Make the persistent network keys compatible with the Go daemon
The daemon requires its key file to be encoded as ProtoBuf value
with a proper `key scheme` discriminator.
2020-02-19 10:58:10 +02:00
Zahary Karadjov 1b1cd915db
Support ENR boot records with the Go daemon 2020-02-19 10:26:32 +02:00
Zahary Karadjov 052a6ac3bc
Hotfix: disable the bootstrap node connectivity check
This used to behave properly before the rebase, but currently
it forces the bootstrap node to exit, because it ends up being
launched with an ENR list telling it to connect to itself.

The root cause will be investigated in a follow-up PR.
2020-02-18 20:42:25 +02:00
Zahary Karadjov 57811dce89 Don't log unwanted lines in beacon_node --help 2020-02-18 12:53:49 +02:00
Zahary Karadjov b2d5aba749 Advertise the TCP port of the bootstrap node properly 2020-02-18 12:53:49 +02:00
Zahary Karadjov 2a3e40e298 Fix test_discovery_helpers 2020-02-18 12:53:49 +02:00
Zahary Karadjov 2c814db750 Temporary switch to supporting only ENR bootstrap addresses
Turns out the DiscV5 code relies heavily on the presence of ENR
records at the moment, so we cannot drive it with ENodes. @kdeme
is working on refactoring that will relax these requirements.
2020-02-18 12:53:49 +02:00
Zahary Karadjov 7f395e554a WIP Working local network sim with discovery
* The bootstrap_nodes.txt file in the node's data dir is now optional
* Log more data on start-up
* Use the latest ENR APIs
* Fix simulation build errors
2020-02-18 12:53:49 +02:00
Zahary Karadjov d2f59ebab9 Switch to ENR records in the bootstrap_nodes.txt file 2020-02-18 12:53:49 +02:00
Zahary Karadjov ee7c2c9dff Unify the bootstrap nodes handling code
We no longer discriminate between ENR, MultiAddress or ENode
bootstrap records (all of them are remapped to ENodes).

The discovery loop will stochastically try to reconnect to
accidentally disconnected nodes.
2020-02-18 12:53:49 +02:00
Zahary Karadjov 4fb654f2af Remove some SSZ compilation warnings 2020-02-18 12:53:49 +02:00
Zahary Karadjov a620c83916 Handle status bar spanning multiple lines (relevant for the Tmux network sim) 2020-02-18 12:53:49 +02:00
Dustin Brody bde1eefdcf improve option naming 2020-02-17 22:00:41 +00:00
Dustin Brody e13846bec6 allow beacon_node to verify finalization when appropriate, and enable by default for make eth2_network_simulation 2020-02-17 22:00:41 +00:00
Jacek Sieka d99ce1bcf0
fix json compile 2020-02-08 19:42:45 +01:00
Jacek Sieka 521b0ed6ba
Clean 20200205 (#729)
* beacon node code cleanup
* rudimentary error checking on mainnet monitor
* start client even when sending deposit
* work around missing block number exception
* connect to testnet with web3 url
* pretty-print digests in json
2020-02-07 08:13:38 +01:00
tersec 09d735212d
initial refactoring of block sanity test runner, with several new tests (#736)
* initial refactoring of block sanity test runner, with several new tests enabled

* remove trailing whitespace
2020-02-07 08:11:26 +01:00
Jacek Sieka 7efd113a75 store empty slot states also 2020-02-05 19:24:07 +00:00
Jacek Sieka fb9c4fabf4 fix state rewind
* rewind fast path comparison was not taking skipped slots into account
properly
* less messy blockref creation
2020-02-05 19:24:07 +00:00
Dmitriy Ryajov b39f36b49b use floodsub instead of gossipsub 2020-02-05 17:20:04 +01:00
Dustin Brody 1ffc2df23d add a couple new deposit tests; fix the false-positive BLS verifications while keeping all but two tests working, despite mismatched 0.9/0.10 BLS standards; better-factor the skipping of BLS validation and Merkle tree validation 2020-02-04 18:39:38 +00:00
Dustin Brody e4922cb177 implement compute_signing_root() and mark some functions as 0.10.1-compatible 2020-02-04 12:24:30 +00:00
Dustin Brody 45dd12cf3f update process_deposit() to actually check is_valid_merkle_branch() unless skipValidation specified 2020-01-30 09:31:08 +00:00
cheatfate 98dc701473 Add PeerPool.addPeer async version and tests. 2020-01-29 15:28:41 +00:00
cheatfate db20fc1172 Fix SyncQueue push(data) bug.
Rename lastSlot to HeadSlot.
Add failure test.
2020-01-29 15:28:41 +00:00
cheatfate 73dc72583f Initial commit. 2020-01-29 15:28:41 +00:00
Zahary Karadjov 1f1a244f3f Implement more checks and wire up the invalid SSZ tests 2020-01-29 10:10:28 +02:00
Zahary Karadjov 182593ccb5 Tenative fix for the test suite 2020-01-29 10:10:28 +02:00
Zahary Karadjov 2a3c237bbb Handle malformed SSZ inputs properly 2020-01-29 10:10:28 +02:00
Dustin Brody 60b1775879 update check_attestation/process_attestation() to 0.10.1 and fill in missing check 2020-01-28 12:07:33 +02:00
Dustin Brody 3efbe36c30 additional 0.10.1 updates where already-implemented code is already done 2020-01-27 12:50:54 +00:00
Dustin Brody dab107fa1e more 0.10.1 updates 2020-01-27 12:50:54 +00:00
Mamy Ratsimbazafy 1ba2cebd85 v0.10.1 (test vectors + unchanged parts) (#696)
* update to 0.10.1

* SSZ Generic and nbench uses the v0.10.1 fixtures

* Tests + spec links: v0.10.0 -> v0.10.1

* Add v0.10.1 TODO in get_latest_attesting_balance (forkchoice)

* SSZ Bytes are now ByteList

* Remove nim-result submodules that was leftover/added by mistake in the branch
2020-01-27 10:56:32 +00:00
Jacek Sieka 270c25c8b8
drop rlpx support (#679) 2020-01-24 09:32:52 +01:00
Jacek Sieka 7dc2434e0d
clean up attestation logging 2020-01-23 18:48:26 +01:00
Ștefan Talpalaru 9cade2a5c4 LMDB: fix comment 2020-01-23 18:23:27 +02:00
Jacek Sieka d9c98b15c8 fix block pool finalization point init 2020-01-23 14:34:58 +02:00
Jacek Sieka 95437e103a disable state pruning
* fix crash when state root is present but state is missing
* fix state root removal when state is removed
* fix block pool initialization which needs tail state
* remove tail block pruning
  * incomplete - fork states are not pruned
  * incomplete - fork blocks are not pruned
  * incomplete - empty slot states are not pruned
  * unknown - tail/finalized block on empty slot might be incorrect
2020-01-22 19:45:38 +02:00