Commit Graph

102 Commits

Author SHA1 Message Date
Jacek Sieka de779c7812 update validator metrics on startup 2020-12-16 19:42:19 +02:00
Jacek Sieka 7d5edb4353
use new stew helpers for assignment (#2172)
* bump libp2p (reduces libp2p gossip memory usage to ~1/3)
* use "generic" assign version
2020-12-16 09:37:22 +01:00
tersec 8d1443f03c
detect already-aggregate-voted condition before attestation pool; add is_aggregator tests (#2170)
* detect already-aggregate-voted condition before attestation pool; add is_aggregator tests

* replace pair of attestation-per-epoch tracking lists with single list and remove Option use

* fix attestation condition

* use safer type conversions; add more is_aggregator tests
2020-12-14 20:58:32 +00:00
tersec 72d00e33b8
always wait for 2/3 into slot for aggregate attestations (#2158) 2020-12-07 20:32:34 +01:00
tersec 26ea76bbdf
don't lag aggregated attestations by a slot (#2150)
* don't lag aggregated attestations by a slot

* don't use aggregation topic at all

* use aggregates again, but with aggressively low ATTESTATION_PROPAGATION_SLOT_RANGE; seems to hold on to LH 1.0 nodes

* clean up scaffolding and double ATTESTATION_PROPAGATION_SLOT_RANGE to 16

* increase ATTESTATION_PROPAGATION_SLOT_RANGE to 24

* increase ATTESTATION_PROPAGATION_SLOT_RANGE to 28 and isolate in only used function due to customization; remove TRAILING_DISTANCE machinery
2020-12-07 12:56:49 +01:00
cheatfate 7e4ff7a740 Implementation of post_v1_beacon_pool_attester_slashings() and post_v1_beacon_pool_proposer_slashings().
Remove doAssert.
2020-12-03 11:40:08 +02:00
Zahary Karadjov 4eaf29875a Less risk for Eth1-induced delays in block proposal 2020-12-01 15:38:50 +02:00
Zahary Karadjov ac9bdde543
Don't rely on a metric value for the ETH display in the status bar 2020-11-29 23:35:39 +02:00
Zahary Karadjov 3c0dfc2fbe Implement the 'deposits exit' command; Remove 'deposits create' 2020-11-29 23:08:07 +02:00
Zahary Karadjov bf2673abc4
Restore the ETH display in the status bar 2020-11-28 20:53:51 +02:00
Jacek Sieka 8685eb7042 add validator balance metrics 2020-11-28 16:22:09 +02:00
zah 372c9b798c
Fix the corrupted database state on Pyrmont nodes; Add mainnet genesis (#2056)
* Handle some web3 timeouts better

* Add support for developer .env files

* Eth1 improvements; Mainnet genesis state

Notable changes:

* The deposits table have been removed from the database. The client
  will no longer process all deposits on start-up.

* The network metadata now includes a "state snapshot" of the deposit
  contract. This allows the client to skip syncing deposits made prior
  to the snapshot (i.e. genesis). Suitable metadata added for Pyrmont
  and Mainnet.

* The Eth1 monitor won't be started unless there are validators attached
  to the node.

* The genesis detection code is now optional and disabled by default

* Bugfix: The client should not produce blocks that will fail validation
  when it hasn't downloaded the latest deposits yet

* Bugfix: Work around the database corruption affecting Pyrmont nodes

* Remove metadata for Toledo and Medalla
2020-11-24 22:21:47 +01:00
Jacek Sieka dbcc0686ff delay pruning of cache for finalized epoch (fixes #2049) 2020-11-20 20:57:50 +02:00
Zahary Karadjov e22248bca1 Tentative fix for the Pyrmont Eth1 voting issue 2020-11-20 17:13:51 +02:00
tersec 9e716b32bd
address some XDeclaredButNotUsed hints (#2028) 2020-11-17 11:14:53 +01:00
Jacek Sieka a6b188bfd4
misc fixes (#2027)
* log when database is loading (to avoid confusion)
* generate network keys later during startup
* fix quarantine not scheduling chain of parents for download and
increase size to one epoch
* log validator count, enr and peerid more clearly on startup
2020-11-16 20:15:43 +01:00
tersec e2f161dbf7
fix attestation sending schedules to avoid timing attack (#1995) 2020-11-16 10:44:18 +01:00
cheatfate 87c92ea05f Address review comments.
Remove `-8.0` bucket because `-4.0` is a limit.
2020-11-12 21:52:01 +02:00
cheatfate dbb058682c Add beacon_attestation_sent_delay metric. 2020-11-12 21:52:01 +02:00
Zahary Karadjov 389c11743a
Review TODO items and self-assign the most important ones 2020-11-10 20:41:04 +02:00
tersec 271df8b604
bump 1.0.0rc-0 spec refs to 1.0.0 (#1974) 2020-11-09 14:18:55 +00:00
Jacek Sieka 95f5f76180
Datatype cleanup (#1953)
* clear up spec todo

* test fix

* remove unnecessary toSszType

* type

* one more
2020-11-04 21:52:47 +00:00
Zahary Karadjov 6ee80651c7 Rename mainchain_monitor to eth1_monitor 2020-11-03 23:23:10 +02:00
Zahary Karadjov 2f29e3e7f3 Further reduce the number of RPC requests; Significantly faster ETH1 syncing 2020-11-03 23:23:10 +02:00
Zahary Karadjov 18639c3eff Don't require requests that might fail on non-archive Geth nodes 2020-11-03 23:23:10 +02:00
Sacha Saint-Leger 7294d98ed1
Update validator_duties.nim (Improve node out of sync message) (#1933)
* Update validator_duties.nim

Improve node out of sync message

* Update validator_duties.nim
2020-11-02 08:52:16 +01:00
Jacek Sieka ff3f8741de
handle fork choice selecting a head unknown to chaindag (#1907)
* move head slot metric to head selection
* avoid redundant head selection in validator duties
2020-10-28 07:55:36 +00:00
Jacek Sieka 499e5ca991
misc memory and perf fixes (#1899)
* misc memory and perf fixes

* use EpochRef for attestation aggregation
* compress effective balances in memory (medalla unfinalized: 4gb ->
1gb)
* avoid hitting db when rewinding to head or clearance state
* avoid hitting db when blocks can be applied to in-memory state -
speeds up startup considerably
* avoid storing epochref in fork choice
* simplify and speed up beacon block creation flow - avoids state reload
thanks to head rewind optimization
* iterator-based committee and attestation participation help avoid lots
of small memory allocations throughout epoch transition (40% speedup on
epoch processing, for example during startup)

* add constant for threshold
2020-10-22 12:53:33 +02:00
cheatfate 1742a82ca7 Proper fix for writeFile() places.
Comment unused/insecure procedure.
2020-10-15 20:18:33 +03:00
Zahary Karadjov e6320e5881 Address #1584 Don't keep all deposits in memory (persist them to disk) 2020-10-15 20:15:51 +03:00
tersec 3ee2dd8da4
p2p-interface spec ref bump (except non-updated places) (#1862) 2020-10-12 14:37:14 +00:00
tersec 1994ffe5a0
update 130+ spec references from v0.12.3 to v1.0.0-rc1 (#1854) 2020-10-12 08:59:24 +00:00
tersec f08f44b9a2
in exit pool, bundle receive messages into beaconblocks (#1812)
* in exit pool, filter out already-packaged messages; bundle remaining messages into beaconblocks

* filter messages at block construction time

* allow adding up to intended capacity of buffers, beyond per-block limits

* document rationale/design for filtering mechanism
2020-10-07 16:57:21 +00:00
Mamy Ratsimbazafy c92d228ec8
Fix non-split attesting protection (#1804)
* address #1679 - att protection in non-split mode

* AttachedValidator public_key -> pubkey

* Always build with slashing protection

* Remove now unnecessary forwarding impl proc
2020-10-06 10:51:33 +02:00
cheatfate 82228fe471 Fix last createDir. 2020-10-05 22:19:50 +03:00
Mamy Ratsimbazafy 0280d6c73e
Revisiting log levels (#1788)
* Update log level - https://github.com/status-im/nim-beacon-chain/issues/1779 https://github.com/status-im/nim-beacon-chain/issues/1785

* Address review comments

* Document the logging strategy [skip ci]
2020-10-01 20:56:42 +02:00
tersec f96ad87d28
switch another 50+ spec refs from v0.12.2 to v0.12.3 (#1749) 2020-09-25 11:52:50 +00:00
tersec 6398a43cc1
update 120+ beacon_chain and validator spec refs from v0.12.2 to v0.12.3 (#1740) 2020-09-24 19:04:10 +02:00
Mamy Ratsimbazafy 52548f079b
Opt-in Slashing protection + interchange (#1643)
* Slashing protection + interchange initial commit

* Restrict the when UseSlashingProtection dance in other modules

* Integrate slashing tests in other all_tests

* Add attestation slashing protection support

* Add a message that mention if built with/without slashing protection

* no op the initialization proc

* test slashing protection in Jenkins (temp)

* where to configure NIMFLAGS in Jenkins ...

* Jenkins -> ensure Built with slashing protection

* Add slashing protection complete import

* use Opt.get(otherwise)

* Don't use negation in proc name

* Turn slashing protection on by default
2020-09-16 13:30:03 +02:00
Jacek Sieka c76305f824
fix some todo (#1645)
* remove some superfluous gcsafes
* remove getTailState (unused)
* don't store old epochrefs in blocks
* document attestation pool a bit
* remove `pcs =` cruft from log
2020-09-14 14:50:03 +00:00
Viktor Kirilov d9f9949ef0 use a separate process for the private keys (Off by default) - there is a new signing_process binary which loads all validators of the beacon node and the BN dictates through stdin of the signing process what to be signed and when and reads from stdout of the process 2020-09-02 12:47:00 +03:00
tersec 02ddc41960
ignore sqlite WAL journals in git; increase logging priority of attestation/block sending (#1590)
* ignore sqlite WAL journal files in git; switch attestation resolved from info to debug

* promote sent attestations/blocks to notice rather than demote resolved attestations/blocks to debug
2020-08-31 14:34:04 +00:00
Mamy Ratsimbazafy 81788becfc
Fork choice - almost free pruning - fix #1534 (#1535)
* initial - cheaper pruning - addresses  #1534

* Pass tests: update offset when pruning, proper handling of pruned parents

* Use options instead of nil for nilable newHead (finalization passing but rootcause not solved)

* First line of defense against stackoverflow in tests

* Fix compute_delta offset after pruning

* Rebase fix - medalla ready

* Remove Option[BlockRef]
2020-08-26 17:23:34 +02:00
Jacek Sieka 22998fdfd4 avoid double deserialization
When blocks and attestations arrive, they are SSZ-decoded twice: once
for validation and once for processing. This branch enqueues the decoded
block directly for processing, avoiding the second, slow
deserialization.

* move processing of blocks and attestations to queue
* ...and out from beacon_node
* split attestation processing into attestations and aggregates
  * also updates metrics
* clean up logging to better follow the lifetime of gossip: arrival,
validation and processing
* drop attestations and aggregates if there are too many
* try to prioritise blocks and aggregates before single-validator
attestations
2020-08-21 11:46:25 +03:00
Dustin Brody bbc90afa27 fix attestation aggregation broadcasting 2020-08-21 11:32:43 +03:00
Jacek Sieka 46c94a18ba rework epoch cache referencing
* collect all epochrefs in specific blocks to make them easier to find
and to avoid lots of small seqs
* reuse validator key databases more aggressively by comparing keys
* make state cache available from within `withState`
* make epochRef available from within onBlockAdded callback
* integrate getEpochInfo into block resolution and epoch ref logic such
that epochrefs are created when blocks are added to pool or lazily when
needed by a getEpochRef
* fill state cache better from EpochRef, speeding up replay and
validation
* store epochRef in specific blocks to make them easier to find and
reuse
* fix database corruption when state is saved while replaying quarantine
* replay slots fully from block pool before processing state
* compare bls values more smartly
* store epoch state without block applied in database - it's recommended
to resync the node!

this branch will drastically speed up processing in times of long
non-finality, as well as cut memory usage by 10x during the recent
medalla madness.
2020-08-19 10:09:06 +03:00
Jacek Sieka 2a36949913
use epochcache for attesting (#1478) 2020-08-10 15:21:31 +02:00
Jacek Sieka 84a501d1ff
remove one cache, add another (#1449)
* remove one cache, add another

This cache removes the need for rewinding in most attestation validation
flow since the attestations come from one of two epochs and must be
targetting a viable block.

Additionally, it also removes all state caches which are less likely to
be used over-all - more metrics are needed to track the rewinding.

On risk is that when chains don't finalize, we'll have lots of epochrefs
in memory meaning lots of validator key databases, most being exactly
the same. This can be addressed in any number of ways. Some of the
memory usage is mitigated by the fact that we previously had lots of big
state caches and now we're keeping only keys instead.

* cleanups

* doc
2020-08-06 19:48:47 +00:00
Zahary Karadjov b427c7249f More logging and metrics (incoming gossip blocks; outgoing aggregated attestations) 2020-08-05 19:28:35 +03:00
Dustin Brody c142de4b7f be more consistent about pubkeys fed to verify_foo_signature() not being separately initialized, while pubkeys, generally, used for matching purposes, elsewhere explicitly initialized 2020-08-04 23:00:33 +03:00