Commit Graph

172 Commits

Author SHA1 Message Date
Mamy Ratsimbazafy d1cb5b7220
Parallel attestation verification (#2718)
* Add parallel attestation verification

* Update tests, batchVerify doesn't use the threadpool with only single core (nim-blscurve update)

* bump nim-blscurve

* Debug info for failing eth2 test vectors

* remove submodule eth2-testnets

* verbose debugging of make failure on Windows (libbacktrace?)

* Remove CI debug mode

* initialization convention

* Fix new altair tests
2021-09-17 03:13:52 +03:00
Jacek Sieka 7a622e8505
rework spec imports (#2779)
The spec imports are a mess to work with, so this branch cleans them up
a bit to ensure that we avoid generic sandwitches and that importing
stuff generally becomes easier.

* reexport crypto/digest/presets because these are part of the public
symbol set of the rest of the spec types
* don't export `merge` types from `base` - this causes circular deps
* fix circular deps in `ssz/spec_types` - this is the first step in
disentangling ssz from spec
* be explicit about phase0 vs altair - longer term, `altair` will become
the "natural" type set, then merge and so on, so no point in giving
`phase0` special preferential treatment
2021-08-12 13:08:20 +00:00
Eugene Kabanov 3b6f4fab4a
New validator client using REST API. (#2651)
* Initial commit.

* Exporting getConfig().

* Add beacon node checking procedures.

* Post rebase fixes.

* Use runSlotLoop() from nimbus_beacon_node.
Fallback implementation.
Fixes for ETH2 REST serialization.

* Add beacon_clock.durationToNextSlot().
Move type declarations from beacon_rest_api to json_rest_serialization.
Fix seq[ValidatorIndex] serialization.
Refactor ValidatorPool and add some utility procedures.
Create separate version of validator_client.

* Post-rebase fixes.
Remove CookedPubKey from validator_pool.nim.

* Now we should be able to produce attestations and aggregate and proofs.
But its not working yet.

* Debugging attestation sending.

* Add durationToNextAttestation.
Optimize some debug logs.
Fix aggregation_bits encoding.
Bump chronos/presto.

* Its alive.

* Fixes for launch_local_testnet script.
Bump chronos.

* Switch client API to not use `/api` prefix.

* Post-rebase adjustments.

* Fix endpoint for publishBlock().

* Add CONFIG_NAME.
Add more checks to ensure that beacon_node is compatible.

* Add beacon committee subscription support to validator_client.

* Fix stacktrace should be an array of strings.
Fix committee subscriptions should not be `data` keyed.

* Log duration to next block proposal.

* Fix beacon_node_status import.

* Use jsonMsgResponse() instead of jsonError().

* Fix graffityBytes usage.
Remove unnecessary `await`.
Adjust creation of SignedBlock instance.
Remove legacy files.

* Rework durationToNextSlot() and durationToNextEpoch() to use `fromNow`.

* Fix race condition for block proposal and attestations for same slot.
Fix local_testnet script to properly kill tasks on Windows.
Bump chronos and nim-http-tools, to allow connections to infura.io (basic auth).

* Catch services errors.
Improve performance of local_testnet.sh script on Windows.
Fix race condition when attestation producing.

* Post-rebase fixes.

* Bump chronos and presto.

* Calculate block publishing delay.
Fix pkill in one more place.

* Add error handling and timeouts to firstSuccess() template.
Add onceToAll() template.
Add checkNodes() procedure.
Refactor firstSuccess() template.
Add error checking to api.nim calls.

* Deprecated usage onceToAll() for better stability.
Address comment and send attestations asap.

* Avoid unnecessary loop when calculating minimal duration.
2021-07-13 13:15:07 +02:00
tersec b1d5609171
remove false OnBlockAdded dependency on phase0 HashedBeaconState (#2661)
* remove false OnBlockAdded dependency on phase.HashedBeaconState

* introduce altair data types into block_clearance; update some alpha.6 spec refs to alpha.7; add get_active_validator_indices_len ForkedHashedBeaconState wrapper

* switch many modules from using datatypes (with phase0 states/blocks) to datatypes/base (fork-independent); update spec refs from alpha.6 to alpha.7 and remove rm'd G2_POINT_AT_INFINITY

* switch more modules from using datatypes (with phase0 states/blocks) to datatypes/base (fork-independent); update spec refs from alpha.6 to alpha.7

* remove unnecessary phase0-only wrapper of get_attesting_indices(); allow signatures_batch to process either fork; remove O(n^2) nested loop in process_inactivity_updates(); add altair support to getAttestationsforTestBlock()

* add Altair versions of asSigVerified(), asTrusted(), and makeBeaconBlock()

* fix spec URL to be Altair for Altair makeBeaconBlock()
2021-06-21 08:35:24 +00:00
Zahary Karadjov 45430f2904
Fix an accidentally commented out line 2021-06-10 11:42:40 +03:00
Johann Bauer 141c39cfaa
Improve explanation of the --doppelganger-detection option in --help. (#2626)
* Improve explanation of the --doppelganger-detection option in --help.

* Update beacon_chain/conf.nim

Co-authored-by: 0xmiel <sacha.saint-leger@ethereum.org>

Co-authored-by: 0xmiel <sacha.saint-leger@ethereum.org>
2021-06-03 11:20:42 +02:00
Zahary Karadjov ed2f6f753d Allow the custom testnet metadata to specify a path to the genesis file 2021-06-01 15:50:50 +03:00
Zahary Karadjov acee2d3709
Remove an inappropriate merge artifact 2021-05-20 13:55:33 +03:00
Zahary Karadjov dc49a51654
Merge stable into unstable (take 2) 2021-05-20 13:52:09 +03:00
Zahary Karadjov b7aa30adfd
Merge stable into unstable 2021-05-20 13:50:40 +03:00
Zahary Karadjov f6a1f602b4
Handle changes in the latest version of Confutils 2021-05-17 21:42:28 +03:00
Zahary Karadjov b9924214ab
Better error-handling for the slashingdb import/export feature
* Error when specifying an invalid --data-dir (or --validator-dir)
* Error when entering an invalid validator public key (e.g. invalid hex value)
* Warning when attempting to export a validator not present in the local database

Some unnecessary remains of the v1 mode has been removed as well
2021-05-17 21:42:23 +03:00
Jacek Sieka 97f4e1fffe
Db1 cont (#2573)
* Revert "Revert "Upgrade database schema" (#2570)"

This reverts commit 6057c2ffb4.

* ssz: fix loading empty lists into existing instances

Not a problem earlier because we didn't reuse instances

* bump nim-eth

* bump nim-web3
2021-05-17 18:37:26 +02:00
Zahary Karadjov 5c313b958e
Simplify the slashing db import/export CLI 2021-05-17 17:12:03 +03:00
tersec 6057c2ffb4
Revert "Upgrade database schema" (#2570)
This reverts commit 22ddf74752.
2021-05-17 06:34:44 +00:00
Mamy André-Ratsimbazafy dacc508992
slashing import integrated in NBC 2021-05-16 21:48:38 +03:00
Jacek Sieka 22ddf74752 Upgrade database schema
The `kvstore` design we're using now turns out to not be the best way to
use `sqlite` - in particular, there are some significant benefits to
using rowid in certain situations and to keep data in separate tables.

With this branch, there are massive improvements in startup time
(seconds instead of minutes) and state/block storage and pruning times
(milliseconds instead of seconds) - these improvements can in particular
be seen on slow drives and translate directly into better attestation
performance.

* update kvstore to new keyspace design
* remove `DirStoreRef` and the hidden `--state-db-kind` option - this
was an experiment to store large blobs in files, but with the new
kvstore, there's no compelling reason to do so
* remove `DbMap` - unused and would need updating for new keyspace
design
* introduce separate tables for each data type (blocks, states etc)
* remove "WITHOUT ROWID" pessimization for tables with large blobs
* close DbSeq statements explicitly (and earlier)
* store beacon block summaries in separate table, without SSZ
compression and load them all with single query on startup
* stop storing backwards compat full states
* mark genesis beacon block as trusted
* avoid faststreams when loading SSZ data
* remove `DisagreementBehavior` (unused)
2021-05-14 20:05:23 +03:00
Jacek Sieka efdf759cc0
avoid some slashing protection queries (#2528)
This PR reduces the number of database queries for slashing protection
from 5 reads and 1 write to 2 reads and 1 write in the optimistic case.

In the process, it removes user-level support for writing the database
in the version 1 format in order to simplify the code flow, and prevent
code rot. In particular, the v1 format was not covered by any unit tests
and has no advantages over v2. The concrete code to read and write it
remains for now, in particular to support upgrades from v1 to v2.

The branch also removes the use of concepts which doesn't work with
checked exceptions - in particular, this highlights code that both
raises exceptions and returns error codes, which could be cleaned up in
the future.

* Cache internal validator ID
* Rely on unique index to check for trivial duplicate votes
* Combine two surround vote queries into one
* Combine API for checking and registering slashing into single function

The slashing DB is normally not a bottleneck, but may become one with
high attached validator counts.
2021-05-04 15:17:28 +02:00
Chris Hobcroft a3b9990beb
Fixed confusing typo (#2533)
"One or more" appears to be a better description than "One of more"
2021-05-02 17:01:49 +02:00
Zahary Karadjov b0912b8eaf
Remove some unused parameters 2021-04-14 16:15:22 +03:00
cheatfate 934cc453d9 Add "BETA version" note to the `--rest` server command line parameters descriptions. 2021-04-09 21:42:13 +03:00
cheatfate 5885068c63 Rebase and fixes. 2021-04-09 21:42:13 +03:00
cheatfate 80e79aef97 Add TODO comments for missing implementations.
Change default REST port to use 5052 (Lighthouse).
Add missing checks for maximum amount of validator ids.
2021-04-09 21:42:13 +03:00
cheatfate 72695dd62a Rest API initial implementation. 2021-04-09 21:42:13 +03:00
Zahary Karadjov 9776fbfe17
Merge branch 'version-1.1.0' into unstable 2021-04-08 20:50:06 +03:00
Zahary Karadjov 61669f269d Clarify that the deposit cretion tools are intended for testnets 2021-04-08 19:58:22 +03:00
Zahary Karadjov 40859e4cdc Restore the deposits create command 2021-04-08 19:58:22 +03:00
Zahary Karadjov ba59dd85cd Support for fallback web3 providers; Fix resource leaks on Eth1 monitor restarts 2021-04-08 12:46:42 +03:00
Zahary Karadjov 7451c4171f
v1.1.0-rc; Enable v2 doppelganger detection by default 2021-04-07 00:38:47 +03:00
Jacek Sieka beceb060c4
Write state diffs to separate table (and experimentally, files instead of db) (#2460) 2021-04-06 21:56:45 +03:00
Jacek Sieka 8f441588cd
Add a few help defaults, format all of them the same (#2284) (#2467)
No support in confutils, so so this code simply hardcodes it
2021-04-01 12:28:57 +00:00
Jacek Sieka 2695cfa864
EH cleanup (#2455)
almost 100% raises in nimbus-eth2 now!

* fix some rare exception-related crashes in json-rpc
2021-03-26 07:52:01 +01:00
Giovanni Petrantoni 9558946df4
Expose gossip direct peers (#2246)
* Expose gossip direct peers

* direct-peer fixes

* add explicit info about direct peer

* build fixes wip
2021-03-23 15:10:17 +09:00
Mentor Palokaj 895fe4baf1
Make it clear that the default port for ETH2 is 9000 udp/tcp (#2421)
* Add default port info

* Add default port info

* Edit CLI and docs to reflect 9000 port

* Apply suggestions from code review

Co-authored-by: Jacek Sieka <arnetheduck@gmail.com>
2021-03-22 08:22:56 +01:00
Zahary Karadjov 77ee210762
v1.0.10 2021-03-10 23:33:37 +02:00
Zahary Karadjov 196e3ea6b6
Switch to slashing DB v2 by default 2021-03-09 00:47:29 +02:00
kdeme 5f750f84b4 Use setupAddress for better IP and ports configuration 2021-03-06 00:09:09 +02:00
Mamy Ratsimbazafy d47f53cd9d
Reorg (5/5) (#2377)
* Reorg things left into networking and gossip_processing

* time -> beacon_clock

* fix builds
2021-03-05 14:12:00 +01:00
Jacek Sieka 0dbc7162ac startup cleanup
* fix several memory leaks due to temporaries not being reset during
init
* avoid massive main() function with lots of stuff in it
* disable nim-prompt (unused)
* reuse validator pool instance in eth2_processor
* style cleanup
2021-02-22 23:32:54 +02:00
Zahary Karadjov e1d6df1e5d Continue using the V1 Slashing DB by default 2021-02-20 22:46:35 +02:00
Mamy Ratsimbazafy 5daafd480f
Slashing protection updates (#2333)
* Fix slashing protection always try to migrate at startup
* Add CLI option for dual DB
2021-02-19 17:18:17 +02:00
Dustin Brody 67e4a045a3 simplify doppelganger detection to boolean 2021-02-03 20:55:33 +02:00
Kim De Mey 73d9c2aa53
Add enr auto update cli option and bump nim-eth (#2278) 2021-02-02 09:07:21 +01:00
tersec 6141286547
rename doppelganger protection to doppelganger detection; switch default from warn to stop (#2281) 2021-02-01 12:18:16 +01:00
Zahary Karadjov fa99c3b417 Fix #2261
Also bumps Confutils to allow setting the hidden --web3-mode param
(to allow testing the eth1 syncing without validators)
2021-01-30 01:32:20 +02:00
Dustin Brody 281853eee8 rename options and internal structures to doppelgangerFoo and remove probing 2021-01-30 00:17:54 +02:00
Dustin Brody a16f5afcd5 pre-emptive duplicate validator detection heuristic 2021-01-21 16:03:02 +02:00
Zahary Karadjov c8c819359c
More clear error message when a validator exit was rejected 2021-01-15 19:40:05 +02:00
tersec dde973e2d4
allow always-on subscription to all attestation subnets when gossiping (#2225)
* allow always-on subscription to all attestation subnets when gossiping

* in subscribe-all-subnets mode, consider all subnets to be stability subnets for ENR purposes
2021-01-12 13:43:15 +01:00
Zahary Karadjov 983b3c9fbf Add a 'we3 test' command for verifying the compatibility of a web3 provider 2020-12-10 02:54:58 +02:00