Commit Graph

2489 Commits

Author SHA1 Message Date
tersec 002bdb834d cleanly close kvstore databases and bump nim-eth (#1630)
* cleanly close kvstore databases

* close databases for all subcommands and during error conditions
2020-09-16 19:21:59 +02:00
Kim De Mey 87e3ef7c56 Bump nim-eth to use lrucache for discovery sessions (#1622) 2020-09-16 19:21:59 +02:00
Jacek Sieka bcbc7acd18 remove chronos future tracking from default build (#1628)
uses a lot of memory, specially during sync
2020-09-16 19:21:59 +02:00
tersec c08b786e77 add ncli_db subcommand to prune database of unnecessary blocks and states (#1593)
* add ncli_db subcommand to prune database of unnecessary blocks, states, and state roots

* tweak comments

* reduce default aggressiveness in pruning old states

* move copyPrunedDatabase() to ncli_db, as it's not generally useful as part of beacon_chain_db and doesn't use any internal interfaces
2020-09-16 19:21:59 +02:00
Eugene Kabanov dcdb5dfade Syncing V2 (#1602)
* Syncing workers now not bound to peers.
Sync status is now printed in statusbar.

* Add `SyncQueue.outSlot` to statusbar too.

* Add `inRangeEvent` and `rangeAge` parameter.

* Fix rangeAge is not depends on SyncQueue latest slot.
Fix syncManager to start from latest local head slot.

* Add notInRange event.

* Remove suspects field.
2020-09-16 19:21:59 +02:00
Jacek Sieka 1d3e2935f2 await validator duties (#1626)
Validator duties proceed slot-by-slot - we should not start a new
validator duty iteration before the previous one is gone or we might run
into consistency and voting issues
2020-09-16 19:21:59 +02:00
Jacek Sieka 04b49cae35 Quick fix to prune some states, pending smarter state storage (#1624)
* Quick fix to prune some states, pending smarter state storage

Adverse effects might include slow rewinds - typically the protocol
doesn't ask for pre-finalized states but RPC might

* document issue, add test

* fix cache miss log
2020-09-16 19:21:59 +02:00
Eugene Kabanov de238c62b7 Bump chronos to help investigate Index problem. (#1625) 2020-09-16 19:21:59 +02:00
Jacek Sieka 07e7916b30 fix req/resp protocol (#1621)
per spec, we must half-close request stream - not doing so may lead to
failure of the other end to start processing our request leading to
timeouts.

In particular, this fixes many sync problems that have been seen on
medalla.

* remove safeClose - close no longer raises
* use per-chunk timeouts in request processing
2020-09-16 19:21:59 +02:00
Sacha Saint-Leger 67ba595ce4 fix low peer count command typo (#1619) 2020-09-16 19:21:59 +02:00
Jacek Sieka 6bd0b760ad bump libp2p, introduce AsyncQueue BufferStream (#1599)
* Syncing workers now not bound to peers.
Sync status is now printed in statusbar.

* Add `SyncQueue.outSlot` to statusbar too.

* Add `inRangeEvent` and `rangeAge` parameter.

* Fix rangeAge is not depends on SyncQueue latest slot.
Fix syncManager to start from latest local head slot.

* Add notInRange event.

* bump libp2p, introduce AsyncQueue BufferStream

* bump

Co-authored-by: cheatfate <eugene.kabanov@status.im>
2020-09-16 19:21:59 +02:00
tersec 78ec548313 use V=1 in Jenkinsfile (#1618) 2020-09-16 19:21:59 +02:00
tersec 3a603d303f match log filenames more precisely for logtrace analysis (#1617)
* match log filenames more precisely for logtrace analysis
2020-09-16 19:21:59 +02:00
Dustin Brody 0c5cae701e remove obsolete comment 2020-09-16 19:21:59 +02:00
tersec cb1a5cf2ca Fix some warnings and hints and partly revert #1610 (#1615)
* address some XDeclaredButNotUsed, ConvFromXtoItselfNotNeeded, and UnusedImport hints and warnings

* partly revert #1610
2020-09-16 19:21:59 +02:00
tersec 45dd971955 mark comments documenting functions as a whole as such (#1613) 2020-09-16 19:21:59 +02:00
Jacek Sieka 7e3f20236f better state cache reuse (#1612) 2020-09-16 19:21:59 +02:00
tersec 1f9cd8ce2b stop discarding future epochs; remove a StateCache() construction (#1610)
* stop discarding non-existent future epochs during epoch state transitions; remove a pointless StateCache() construction in advance_slots()

* update nbench to pass StateCache to process_slots()
2020-09-16 19:21:59 +02:00
Jacek Sieka 1b16ad10b1 bump libp2p (#1609) 2020-09-16 19:21:59 +02:00
Jacek Sieka 625e0912de simplify libp2p logging (#1605)
and a few other small logging fixes
2020-09-16 19:21:59 +02:00
nbc-bump-bot[bot] 5a55ca8f47 auto-bump nim-libp2p (#1604)
Co-authored-by: = <arnetheduck@users.noreply.github.com>
2020-09-16 19:21:59 +02:00
Jacek Sieka a095a1e24d bump chronos (#1598)
* bump chronos

this reverts backwards-incompatible changes to CancelledError and
resores other functionality.

chronos metrics are lost and will need to be reimplemented.

* bump
2020-09-16 19:21:59 +02:00
tersec d69ab20d47 address issue #1552 (#1601) 2020-09-16 19:21:59 +02:00
Jacek Sieka 29e56232a7 avoid hash tree root calculation when loading blocks from database (#1572) 2020-09-16 19:21:59 +02:00
Ștefan Talpalaru c2384246bc support spaces in $(MAKE) (#1603) 2020-09-16 19:21:59 +02:00
tersec 368bd7c057 address issue #1580 (#1600)
* address issue #1580

* Update beacon_chain/spec/beaconstate.nim

Co-authored-by: Jacek Sieka <jacek@status.im>
2020-09-16 19:21:59 +02:00
Viktor Kirilov bb83817c2a use a separate process for the private keys (Off by default) - there is a new signing_process binary which loads all validators of the beacon node and the BN dictates through stdin of the signing process what to be signed and when and reads from stdout of the process 2020-09-16 19:21:59 +02:00
Viktor Kirilov 96e1a5d70e 50/50 bn/vc split for the validator keys ON by default for the testnet scripts 2020-09-16 19:21:59 +02:00
nbc-bump-bot[bot] 4a2ac2b46a auto-bump nim-libp2p (#1484)
Co-authored-by: = <arnetheduck@users.noreply.github.com>
2020-09-16 19:21:59 +02:00
Viktor Kirilov 9a8e81ceec added the --network=<x> option to the tools for which it matters 2020-09-16 19:21:59 +02:00
Ștefan Talpalaru 3bb2bcdfc9 remove template expansion file from Git control (#1589)
because it changes by simply building the software, creating problems
for beta testers
2020-09-16 19:21:59 +02:00
Jacek Sieka 174e21eb17 perform slashing check before mutation (#1594)
avoid copy of validator as well
2020-09-16 19:21:59 +02:00
tersec 1fefe2bb19 ignore sqlite WAL journals in git; increase logging priority of attestation/block sending (#1590)
* ignore sqlite WAL journal files in git; switch attestation resolved from info to debug

* promote sent attestations/blocks to notice rather than demote resolved attestations/blocks to debug
2020-09-16 19:21:59 +02:00
tersec a52b0436fd bound block quarantine size (#1564)
* bound block quarantine size

* add additional logging for block quarantining

* re-add quarantine.add() call

* remove pre-finalization blocks; add logging for full quarantine

* clear quarantine on chain reorganization

* update block_sim and tests

* update test_attestation_pool
2020-09-16 19:21:59 +02:00
tersec 0a4bbf204b bump nim-eth (#1579) 2020-09-16 19:21:59 +02:00
Zed 3574d1b621 Makefile: add clean-cross task for cross compiling (#1554) 2020-09-16 19:21:59 +02:00
Sacha Saint-Leger 24dae4b04f update troubleshooting medalla (#1578) 2020-09-16 19:21:59 +02:00
Ștefan Talpalaru 76675fc83e fix medalla-dev 2020-09-16 19:21:59 +02:00
Ștefan Talpalaru 80a50d6389 remove BUILD_LOG_LEVEL 2020-09-16 19:21:59 +02:00
tersec 0e6619da69 add setting for benchmarking and profiling of sqlite block storage times (#1575) 2020-09-16 19:21:59 +02:00
Jacek Sieka 2aa9f3e17b log getblocks error 2020-09-16 19:21:59 +02:00
Jacek Sieka 6202ede3d9 implement clock disparity for attestation validation (#1568)
This implements disparity, resolving a part of
https://github.com/status-im/nim-beacon-chain/issues/1367

* make BeaconTime a duration for fractional seconds
* factor out attestation/aggregate validation
* simplify recording of queued attestations
* simplify attestation signature check
* fix blocks_received metric
* add some trivial validation tests
* remove unresolved attestation table - attestations for unknown blocks
are dropped instead (cannot verify their signature)
2020-09-16 19:21:59 +02:00
Jacek Sieka a0bff42016 avoid unnecessary seq allocations (#1573)
"sigh"
2020-09-16 19:21:59 +02:00
Zahary Karadjov 0e64b4e3b3 Update the Nimbus bootstrap nodes 2020-09-16 19:21:59 +02:00
Sacha Saint-Leger 425823ea54 Medalla guide updates + troubleshooting page (#1571)
* book updates

* troubleshooting: introduce devel branch, and link to discord

* add disclaimer and recommendation to fall back to master

* minor edits
2020-09-16 19:21:59 +02:00
Ștefan Talpalaru 484ab55ec0 Jenkins: run benchmarks (#1524)
* Jenkins: run benchmarks

* don't exit the workspace

* try calling the benchmark plugin from the config file

* pass all params

* separate Jenkins job for benchmarks
2020-09-16 19:21:59 +02:00
tersec d0ae51e8a2 harden beacon_pending_deposits metrics calculation against overflow (#1566)
* harden beacon_pending_deposits metrics calculation

* ...

* move beacon_pending_deposits and beacon_processed_deposits_total out of specs and into chain DAG
2020-09-16 19:21:59 +02:00
Eugene Kabanov 5d744066e4 Request manager ignores non-critical errors while processing blocks. (#1569)
* Request manager ignores non-critical errors while processing blocks. Only BlockError.Invalid become critical error.

* Add some comments.
2020-09-16 19:21:59 +02:00
Mamy Ratsimbazafy a29faadace Fork choice - almost free pruning - fix #1534 (#1535)
* initial - cheaper pruning - addresses  #1534

* Pass tests: update offset when pruning, proper handling of pruned parents

* Use options instead of nil for nilable newHead (finalization passing but rootcause not solved)

* First line of defense against stackoverflow in tests

* Fix compute_delta offset after pruning

* Rebase fix - medalla ready

* Remove Option[BlockRef]
2020-09-16 19:21:59 +02:00
Jacek Sieka 1ef98c36ac reuse validator key cache better (#1562)
new key cache can be used for old epochs in the same tree
2020-09-16 19:21:59 +02:00