1358 Commits

Author SHA1 Message Date
Eugene Kabanov
6e463257f4
PeerPool fixes. (#1654)
* Refactor peer_pool.
Fix eth2_network peer counters.
Fix PeerPool do not allow to add more peers when empty space available.

* Remove unused imports.

* Add test for a bug.

* Fix eth2_network disconnect should deletePeer not release.
More PeerPool refactoring.
2020-09-16 13:00:11 +03:00
tersec
6d8130dc49
close block_sim database; remove code duplication in exit_pool (#1656) 2020-09-16 09:16:23 +02:00
Eugene Kabanov
9abdbdabd8
Fix sync_manager.nim not rewarding peers for good responses. (#1660) 2020-09-16 09:15:06 +02:00
tersec
7e10b9850b
enable topic unsubscribing and attestation subnet cycling (#1646)
* enable topic unsubscribing and attestation subnet cycling

* remove refences to states

* waitFor -> await

* revert exit pool changes for a cleaner PR
2020-09-15 12:40:43 +00:00
Jacek Sieka
c76305f824
fix some todo (#1645)
* remove some superfluous gcsafes
* remove getTailState (unused)
* don't store old epochrefs in blocks
* document attestation pool a bit
* remove `pcs =` cruft from log
2020-09-14 14:50:03 +00:00
tersec
9f21bbd666
[WIP] skeleton of attester slashing pool & validators (#1639)
* skeleton of attester slashing pool & validators

* add skeleton for proposer slashings and voluntary exits; rename pool to more inclusive exit pool to stay consistent with all three; ensure is initialized by beacon_node so is safe to merge, even if it doesn't do much yet
2020-09-14 16:26:31 +02:00
Ștefan Talpalaru
60b8905852
reorder startup steps (#1642)
so the Ctr+C hook and the metrics server go before the potentially slow
`BeaconNode.init()`
2020-09-14 08:24:31 +02:00
tersec
aca1a318f2
cleanly close kvstore databases and bump nim-eth (#1630)
* cleanly close kvstore databases

* close databases for all subcommands and during error conditions
2020-09-12 05:35:58 +00:00
Kim De Mey
5fff800cf8
Bump nim-eth to use lrucache for discovery sessions (#1622) 2020-09-11 17:46:48 +00:00
tersec
48893f1c2e
add ncli_db subcommand to prune database of unnecessary blocks and states (#1593)
* add ncli_db subcommand to prune database of unnecessary blocks, states, and state roots

* tweak comments

* reduce default aggressiveness in pruning old states

* move copyPrunedDatabase() to ncli_db, as it's not generally useful as part of beacon_chain_db and doesn't use any internal interfaces
2020-09-11 15:20:34 +02:00
Eugene Kabanov
c7c9b9d5f1
Syncing V2 (#1602)
* Syncing workers now not bound to peers.
Sync status is now printed in statusbar.

* Add `SyncQueue.outSlot` to statusbar too.

* Add `inRangeEvent` and `rangeAge` parameter.

* Fix rangeAge is not depends on SyncQueue latest slot.
Fix syncManager to start from latest local head slot.

* Add notInRange event.

* Remove suspects field.
2020-09-11 14:46:01 +02:00
Jacek Sieka
775683cf2e
await validator duties (#1626)
Validator duties proceed slot-by-slot - we should not start a new
validator duty iteration before the previous one is gone or we might run
into consistency and voting issues
2020-09-11 09:50:00 +00:00
Jacek Sieka
8a5a261fcd
Quick fix to prune some states, pending smarter state storage (#1624)
* Quick fix to prune some states, pending smarter state storage

Adverse effects might include slow rewinds - typically the protocol
doesn't ask for pre-finalized states but RPC might

* document issue, add test

* fix cache miss log
2020-09-11 10:03:50 +02:00
Jacek Sieka
a087909683
fix req/resp protocol (#1621)
per spec, we must half-close request stream - not doing so may lead to
failure of the other end to start processing our request leading to
timeouts.

In particular, this fixes many sync problems that have been seen on
medalla.

* remove safeClose - close no longer raises
* use per-chunk timeouts in request processing
2020-09-10 21:40:09 +02:00
tersec
d0de1a49a3
Fix some warnings and hints and partly revert #1610 (#1615)
* address some XDeclaredButNotUsed, ConvFromXtoItselfNotNeeded, and UnusedImport hints and warnings

* partly revert #1610
2020-09-08 11:32:43 +00:00
tersec
b3b578501a
mark comments documenting functions as a whole as such (#1613) 2020-09-08 08:54:55 +00:00
Jacek Sieka
dde26f359f
better state cache reuse (#1612) 2020-09-08 09:23:48 +02:00
tersec
3d5f24f14c
stop discarding future epochs; remove a StateCache() construction (#1610)
* stop discarding non-existent future epochs during epoch state transitions; remove a pointless StateCache() construction in advance_slots()

* update nbench to pass StateCache to process_slots()
2020-09-07 15:04:33 +00:00
Jacek Sieka
d584591ded
simplify libp2p logging (#1605)
and a few other small logging fixes
2020-09-06 10:39:25 +02:00
tersec
456bdc87cd
address issue #1552 (#1601) 2020-09-04 08:39:46 +02:00
Jacek Sieka
aed57df957
avoid hash tree root calculation when loading blocks from database (#1572) 2020-09-04 08:35:10 +02:00
tersec
e4a43f7628
address issue #1580 (#1600)
* address issue #1580

* Update beacon_chain/spec/beaconstate.nim

Co-authored-by: Jacek Sieka <jacek@status.im>
2020-09-02 16:16:25 +00:00
Viktor Kirilov
d9f9949ef0 use a separate process for the private keys (Off by default) - there is a new signing_process binary which loads all validators of the beacon node and the BN dictates through stdin of the signing process what to be signed and when and reads from stdout of the process 2020-09-02 12:47:00 +03:00
Viktor Kirilov
65d7787b1e 50/50 bn/vc split for the validator keys ON by default for the testnet scripts 2020-09-01 16:39:07 +03:00
Viktor Kirilov
67d73c4c60 added the --network=<x> option to the tools for which it matters 2020-09-01 12:02:22 +03:00
Ștefan Talpalaru
43c6a3634b
remove template expansion file from Git control (#1589)
because it changes by simply building the software, creating problems
for beta testers
2020-09-01 09:59:19 +02:00
Jacek Sieka
124ec4b737
perform slashing check before mutation (#1594)
avoid copy of validator as well
2020-09-01 09:49:55 +02:00
tersec
02ddc41960
ignore sqlite WAL journals in git; increase logging priority of attestation/block sending (#1590)
* ignore sqlite WAL journal files in git; switch attestation resolved from info to debug

* promote sent attestations/blocks to notice rather than demote resolved attestations/blocks to debug
2020-08-31 14:34:04 +00:00
tersec
ab255662df
bound block quarantine size (#1564)
* bound block quarantine size

* add additional logging for block quarantining

* re-add quarantine.add() call

* remove pre-finalization blocks; add logging for full quarantine

* clear quarantine on chain reorganization

* update block_sim and tests

* update test_attestation_pool
2020-08-31 11:00:38 +02:00
Jacek Sieka
c810b64ed8
log getblocks error 2020-08-27 10:24:41 +02:00
Jacek Sieka
fa1621db46
implement clock disparity for attestation validation (#1568)
This implements disparity, resolving a part of
https://github.com/status-im/nim-beacon-chain/issues/1367

* make BeaconTime a duration for fractional seconds
* factor out attestation/aggregate validation
* simplify recording of queued attestations
* simplify attestation signature check
* fix blocks_received metric
* add some trivial validation tests
* remove unresolved attestation table - attestations for unknown blocks
are dropped instead (cannot verify their signature)
2020-08-27 09:34:12 +02:00
Jacek Sieka
2081c4f505
avoid unnecessary seq allocations (#1573)
"sigh"
2020-08-27 08:32:51 +02:00
tersec
83667dddfe
harden beacon_pending_deposits metrics calculation against overflow (#1566)
* harden beacon_pending_deposits metrics calculation

* ...

* move beacon_pending_deposits and beacon_processed_deposits_total out of specs and into chain DAG
2020-08-26 17:25:39 +02:00
Eugene Kabanov
b525dc9450
Request manager ignores non-critical errors while processing blocks. (#1569)
* Request manager ignores non-critical errors while processing blocks. Only BlockError.Invalid become critical error.

* Add some comments.
2020-08-26 17:24:59 +02:00
Mamy Ratsimbazafy
81788becfc
Fork choice - almost free pruning - fix #1534 (#1535)
* initial - cheaper pruning - addresses  #1534

* Pass tests: update offset when pruning, proper handling of pruned parents

* Use options instead of nil for nilable newHead (finalization passing but rootcause not solved)

* First line of defense against stackoverflow in tests

* Fix compute_delta offset after pruning

* Rebase fix - medalla ready

* Remove Option[BlockRef]
2020-08-26 17:23:34 +02:00
Jacek Sieka
f26d6a4fd3
reuse validator key cache better (#1562)
new key cache can be used for old epochs in the same tree
2020-08-26 17:06:40 +02:00
Ștefan Talpalaru
ddd8489de0
bump vendor/nim-blscurve (#1560)
and log the BLS backend
2020-08-24 14:56:45 +02:00
Jacek Sieka
a7a279d615
add option to disable discv5 (#1509) 2020-08-24 13:52:06 +02:00
Dustin Brody
95d5736128 don't rely on head updates for topic subscription decision 2020-08-22 01:50:50 +03:00
Dustin Brody
cac4fee569 remove waitFor 2020-08-22 01:50:50 +03:00
Dustin Brody
d9bf6f450d delay starting gossipsub until near-synchronized 2020-08-22 01:50:50 +03:00
Zahary Karadjov
8809f8d200 Implement 'wallets restore' and 'wallets list'; Update 'nextAccount' properly after making deposits multiple times 2020-08-22 00:16:18 +03:00
Zahary Karadjov
21b5ad3259
make clearn-medalla will preserve your validator files 2020-08-21 21:50:46 +03:00
Dustin Brody
99e330d014 fix underflow in deposit procesing (#1542) 2020-08-21 19:00:36 +03:00
Jacek Sieka
61538fa581 speed up shuffling
Replace shuffling function with zrnt version - `get_shuffled_seq` in
particular puts more strain on the GC by allocating superfluous seq's
which turns out to have a significant impact on block processing (when
replaying blocks for example) - 4x improvement on non-epoch, 1.5x on
epoch blocks (replay is done without signature checking)

Medalla, first 10k slots - pre:

```
Loaded 68973 blocks, head slot 117077
All time are ms
Average,       StdDev,          Min,          Max,      Samples,
Test
Validation is turned off meaning that no BLS operations are performed
76855.848,        0.000,    76855.848,    76855.848,            1,
Initialize DB
1.073,        0.914,        0.071,       12.454,         7831,
Load block from database
31.382,        0.000,       31.382,       31.382,            1,
Load state from database
85.644,       30.350,        3.056,      466.136,         7519,
Apply block
506.569,       91.129,      130.654,      874.786,          312,
Apply epoch block
```

post:

```
Loaded 68973 blocks, head slot 117077
All time are ms
Average,       StdDev,          Min,          Max,      Samples,
Test
Validation is turned off meaning that no BLS operations are performed
72457.303,        0.000,    72457.303,    72457.303,            1,
Initialize DB
1.015,        0.858,        0.070,       11.231,         7831,
Load block from database
28.983,        0.000,       28.983,       28.983,            1,
Load state from database
21.725,       17.461,        2.659,      393.217,         7519,
Apply block
324.012,       33.954,       45.452,      440.532,          312,
Apply epoch block
```
2020-08-21 16:05:10 +03:00
cheatfate
5fc07fef75 Workaround fix password issues on Windows. 2020-08-21 12:55:49 +03:00
Viktor Kirilov
678a7efaaa moved away from WithState() for the common validator duties in the API - using EpochRef 2020-08-21 11:47:43 +03:00
Jacek Sieka
22998fdfd4 avoid double deserialization
When blocks and attestations arrive, they are SSZ-decoded twice: once
for validation and once for processing. This branch enqueues the decoded
block directly for processing, avoiding the second, slow
deserialization.

* move processing of blocks and attestations to queue
* ...and out from beacon_node
* split attestation processing into attestations and aggregates
  * also updates metrics
* clean up logging to better follow the lifetime of gossip: arrival,
validation and processing
* drop attestations and aggregates if there are too many
* try to prioritise blocks and aggregates before single-validator
attestations
2020-08-21 11:46:25 +03:00
Dustin Brody
bbc90afa27 fix attestation aggregation broadcasting 2020-08-21 11:32:43 +03:00
Jacek Sieka
9244ae7a38 more speedups
* evaluate block attestations under the epochref of the block - this is
what the state transition function does
* avoid copying attestation seq unnecessarily
* avoid unnecessary hashset for unslashed indices
2020-08-19 14:51:04 +03:00