Commit Graph

56 Commits

Author SHA1 Message Date
tersec cd087b9a43
replace `optimisticRoots` table with field in `BlockRef` (#4969)
* replace optimisticRoots table with field in BlockRef

* copyright year

* mark finalized blocks as verified on load

* Update beacon_chain/consensus_object_pools/block_dag.nim

Co-authored-by: Etan Kissling <etan@status.im>

* expand non-optimistic block checking to all pre-merge blocks; refactor markBlockVerified to use BlockRef rather than block root and remove superfluous caller in newPayload path replaced by addResolvedHeadBlock BlockRef construction

* don't treat finalized block specially; VALID status is sticky

---------

Co-authored-by: Etan Kissling <etan@status.im>
2023-05-20 12:18:51 +00:00
tersec 74511f61d1
Use withdrawal credentials as default fee recipient (#4968) 2023-05-17 07:56:37 +03:00
zah 5bf9284e62
Initial public version of the Verifying Web3Signer functionality (#4912)
* Allow the list of proved properties for web3signer to be configured
* Document the Web3Signer setups (regular, distributed and verified)
2023-05-09 11:16:43 +03:00
zah 9b2c07c118
Allow exiting multiple validators at once (#4855) 2023-04-25 09:44:01 +03:00
Eugene Kabanov 0ff86e9538
web3signer refactoring and test suite. (#4775)
* Refactor nimbus_signing_node to support Unix signals.

* Fix SN unable to close REST server properly.

* Fix `keys`, `deposit` and `validator_registration` endpoints issues.
Add getValidatorExitSignature() and getDepositMessageSignature() to validator_pool.

* Add /reload endpoint and implementation.
Fix signData to not cancel `timer`.
Fix validator_pool should clear attachedValidators table.

* Diva protocol enhancement implementation.
2023-04-06 16:16:21 +03:00
cheatfate 596006be08
Workaround for issue #4216. 2023-03-21 23:06:57 +02:00
zah 8771e91d53
Support for driving multiple EL nodes from a single Nimbus BN (#4465)
* Support for driving multiple EL nodes from a single Nimbus BN

Full list of changes:

* Eth1Monitor has been renamed to ELManager to match its current
  responsibilities better.

* The ELManager is no longer optional in the code (it won't have
  a nil value under any circumstances).

* The support for subscribing for headers was removed as it only
  worked with WebSockets and contributed significant complexity
  while bringing only a very minor advantage.

* The `--web3-url` parameter has been deprecated in favor of a
  new `--el` parameter. The new parameter has a reasonable default
  value and supports specifying a different JWT for each connection.
  Each connection can also be configured with a different set of
  responsibilities (e.g. download deposits, validate blocks and/or
  produce blocks). On the command-line, these properties can be
  configured through URL properties stored in the #anchor part of
  the URL. In TOML files, they come with a very natural syntax
  (althrough the URL scheme is also supported).

* The previously scattered EL-related state and logic is now moved
  to `eth1_monitor.nim` (this module will be renamed to `el_manager.nim`
  in a follow-up commit). State is assigned properly either to the
  `ELManager` or the to individual `ELConnection` objects where
  appropriate.

  The ELManager executes all Engine API requests against all attached
  EL nodes, in parallel. It compares their results and if there is a
  disagreement regarding the validity of a certain payload, this is
  detected and the beacon node is protected from publishing a block
  with a potential execution layer consensus bug in it.

  The BN provides metrics per EL node for the number of successful or
  failed requests for each type Engine API requests. If an EL node
  goes offline and connectivity is resoted later, we report the
  problem and the remedy in edge-triggered fashion.

* More progress towards implementing Deneb block production in the VC
  and comparing the value of blocks produced by the EL and the builder
  API.

* Adds a Makefile target for the zhejiang testnet
2023-03-05 01:40:21 +00:00
Jacek Sieka 83f9745df1
restore doppelganger check on connectivity loss (#4616)
* restore doppelganger check on connectivity loss

https://github.com/status-im/nimbus-eth2/pull/4398 introduced a
regression in functionality where doppelganger detection would not be
rerun during connectivity loss. This PR reintroduces this check and
makes some adjustments to the implementation to simplify the code flow
for both BN and VC.

* track when check was last performed for each validator (to deal with
late-added validators)
* track when we performed a doppel-detectable activity (attesting) so as
to avoid false positives
* remove nodeStart special case (this should be treated the same as
adding a validator dynamically just after startup)

* allow sync committee duties in doppelganger period

* don't trigger doppelganger when registering duties

* fix crash when expected index response is missing

* fix missing slashingSafe propagation
2023-02-20 13:28:56 +02:00
Eugene Kabanov e91415662b
Keystore cache implementation. (#4372) 2023-02-16 19:25:48 +02:00
zah ff464e49cf
Implement the set of gas_limit end-points in the Keymanager API (#4612)
Fixes #3946
2023-02-15 15:10:31 +00:00
Jacek Sieka 856fcea8d7
fix slow checking of unknown validators (#4590)
We do a linear scan of all pubkeys for each validator and slot - this
becomes expensive with large validator counts.

* normalise BN/VC validator startup logging
* fix crash when host cannot be resolved while adding remote validator
* silence repeated log spam for unknown validators
* print pubkey/index/activation mapping on startup/validator
identification
2023-02-07 14:53:36 +00:00
tersec aacc8d702d
remove Nim 1.2-compatible `push raise`s and update copyright notice years (#4528) 2023-01-20 14:14:37 +00:00
Zahary Karadjov 46fc5716a4
Faster keystore generation in the local sim 2023-01-13 12:31:39 +02:00
Jacek Sieka ba3db7aa5a
spec: Option -> Opt (#4488) 2023-01-11 12:29:21 +00:00
Jacek Sieka 6e2a02466e
unify bn/vc doppelganger detection (#4398)
* fix REST liveness endpoint responding even when gossip is not enabled
* fix VC exit code on doppelganger hit
* fix activation epoch not being updated correctly on long deposit
queues
* fix activation epoch being set incorrectly when updating validator
* move most implementation logic to `validator_pool`, add tests
* ensure consistent logging between VC and BN
* add docs
2022-12-09 17:05:55 +01:00
tersec 031780b612
eip4844 operations tests (#4392) 2022-12-06 12:40:13 +00:00
Eugene Kabanov eb661565ed
Per-validator doppelganger protection. (#4304)
* Initial commit.

* NextAttestationEntry type.

* Add doppelgangerCheck and actual check.

* Recover deleted check.

* Remove NextAttestainEntry changes.

* More cleanups for NextAttestationEntry.

* Address review comments.

* Remove GENESIS_EPOCH specific check branch.

* Decrease number of full epochs for doppelganger check in VC.

Co-authored-by: zah <zahary@status.im>
2022-11-20 15:55:43 +02:00
zah ad63bba446
Support Prysm and Ethdo Keystores (Fixes #4107) (#4149) 2022-09-20 01:09:56 +03:00
Jacek Sieka ef8bab58eb
load suggested fee recipient file also when keymanager is disabled (#4078)
Since these files may have been created in a previous run or manually,
we want to keep loading them even on nodes that don't enable the
keystore API (for example static setups)

Other changes:

* log keystore loading progressively (#3699)
* print initial fee recipient when loading validators
* log dynamic fee recipient updates
2022-09-17 08:30:07 +03:00
zah b1ac9c9fe4
Fix a potential segfault and various potential stalls (#4003)
* Fixes a segfault during block production when the Keymanager API
  is disabled. The Keymanager is now disabled on half of the local
  testnet nodes to catch such problems in the future.

* Fixes multiple potential stalls from REST requests being done
  without a timeout. From practice, we know that such requests
  can hang forever if not cancelled with a timeout. At best,
  this would be a resource leak, at worst, it may lead to a
  full stall of the client and missed validator duties.

* Changes some Options usages to Opt (for easier use of valueOr)
2022-08-19 21:51:30 +00:00
zah fca20e08d6
Keymanager API for the validator client (#3976)
* Keymanager API for the validator client
* Properly treat the 'description' field as optional when loading Keystores
* Spec-compliant serialization of the slashing data in Keymanager's DeleteKeys response ()

Fixes #3940
Fixes #3964
Closes #3884 by adding test
2022-08-19 13:30:07 +03:00
Eugene Kabanov 250f7b4bdf
Exclusive keystore locking (#3907) 2022-08-08 00:53:20 +03:00
Miran dfd4afc9f2
compatibility with Nim 1.4+ (#3888) 2022-07-29 10:53:42 +00:00
Miran 8ebd5a5fe8
fix some warnings (#3886) 2022-07-18 19:17:11 +00:00
Etan Kissling ee4e4ddeab
cleanup `getPersistentNetKeys` for better reuse (#3859)
Allow using `getPersistentNetKeys` without passing a `config`, and reuse
local helpers in its implementation.
2022-07-13 23:26:16 +02:00
zah 806536a040
[Keymanager API] Support for the feerecipient end-points (#3864)
Other changes:

* The Keymanager error responses differ from the Beacon API responses.
  'keymanagerApiError' replaces the former usages of 'jsonError'.

* Return status code 401 and 403 for authorization errors in accordance
  to the spec.

* Eliminate inconsistencies in the REST JSON parsing. Some of the code
  paths allowed missing fields.

* Added logging of serialization failure details at DEBUG level.
2022-07-13 17:45:04 +03:00
Eugene Kabanov 263a2ffa14
Validator client various fixes. (#3840)
* Improve fallback_service.
* Fix nextAction negative time issue.
* Improve logging in fallback_service.
* Improve logging in sync_committee_service.
* Prepare all services for cancellation.
* Signals handlers for validator client
* Address #3800

Co-authored-by: Zahary Karadjov <zahary@gmail.com>
2022-07-13 17:43:57 +03:00
Etan Kissling 2a2bcea70d
group justified and finalized `Checkpoint` (#3841)
The justified and finalized `Checkpoint` are frequently passed around
together. This introduces a new `FinalityCheckpoint` data structure that
combines them into one.

Due to the large usage of this structure in fork choice, also took this
opportunity to update fork choice tests to the latest v1.2.0-rc.1 spec.
Many additional tests enabled, some need more work, e.g. EL mock blocks.
Also implemented `discard_equivocations` which was skipped in #3661,
and improved code reuse across fork choice logic while at it.
2022-07-06 13:33:02 +03:00
Jacek Sieka 347a485b5b
bearssl: split abi (#3755) 2022-06-21 10:29:16 +02:00
zah a0a6dd2f63
Add a ncli tool for converting a regular keystore into a distributed one (#3634) 2022-05-17 16:50:49 +03:00
zah 6d11ad6ce1
Support for distributed keystores with multiple remotes based on threshold signatures (#3616)
Other fixes:

* Fix bit rot in the `make prater-dev-deposit` target.
* Correct content-type in the responses of the Nimbus signing node
* Invalid JSON payload was being sent in the web3signer requests
2022-05-10 03:32:12 +03:00
tersec 4a372410a4
use MAX_CHUNK_SIZE_BELLATRIX for signed Bellatrix blocks (#3613)
* use MAX_CHUNK_SIZE_BELLATRIX for signed Bellatrix blocks

* Update beacon_chain/networking/eth2_network.nim

Co-authored-by: Etan Kissling <etan@status.im>

* localPassC to localPassc

* check against maxChunkSize rather than constant

Co-authored-by: Etan Kissling <etan@status.im>
2022-05-05 05:45:35 +00:00
tersec 28ba2d5544
stylecheck fixes (#3592) 2022-04-14 13:47:14 +03:00
Jacek Sieka f70ff38b53
enable `styleCheck:usages` (#3573)
Some upstream repos still need fixes, but this gets us close enough that
style hints can be enabled by default.

In general, "canonical" spellings are preferred even if they violate
nep-1 - this applies in particular to spec-related stuff like
`genesis_validators_root` which appears throughout the codebase.
2022-04-08 16:22:49 +00:00
zah f89c604fb0
Add Nimble lock file (#3560) 2022-03-30 22:00:03 +00:00
tersec b37bf8c94b
remove TaintedString (#3546)
* remove TaintedString

* dumpDir is used as a helper function for more specific types of dumpDirs
2022-03-24 21:44:34 +00:00
tersec 50f5754e3c
exists{Dir,File} -> {dir,file}Exists; rm unused imports (#3543) 2022-03-24 00:38:48 +00:00
Jacek Sieka 70270eeabe
better error messages on directory creation failure (#3536) 2022-03-22 17:06:21 +00:00
tersec f550eb2f17
fix two typos (#3491) 2022-03-14 12:50:23 +00:00
Eugene Kabanov 40c77e5928
Remote KeyManager API and number of fixes/tests for KeyManager API (#3360)
* Initial commit.

* Fix current test suite.

* Fix keymanager api test.

* Fix wss_sim.

* Add more keystore_management tests.

* Recover deleted isEmptyDir().

* Add `HttpHostUri` distinct type.
Move keymanager calls away from rest_beacon_calls to rest_keymanager_calls.
Add REST serialization of RemoteKeystore and Keystore object.
Add tests for Remote Keystore management API.
Add tests for Keystore management API (Add keystore).
Fix serialzation issues.

* Fix test to use HttpHostUri instead of Uri.

* Add links to specification in comments.

* Remove debugging echoes.
2022-02-07 22:36:09 +02:00
Zahary Karadjov 54d0d588b1 Implementation of the Keymanager API (BETA)
https://github.com/ethereum/keymanager-APIs
2022-01-04 18:51:45 +02:00
Eugene Kabanov e62c7c7c37
Remote signing client/server. (#3077) 2021-11-30 03:20:21 +02:00
Jacek Sieka df3fc9525f
import cleanup (#2997)
* import cleanup

...and remove some unused types

* add random imports

* more imports
2021-10-19 16:09:26 +02:00
zah 29ee9ec81e
Bugfix: don't crash on keystores without description (#2967) 2021-10-07 18:30:34 +02:00
Eugene Kabanov 65257b82f8
Validator key management API (#2755)
Implements https://github.com/ethereum/beacon-APIs/pull/151
2021-10-04 22:08:31 +03:00
Jacek Sieka 6d47d96c84
altair upgrade for prater (#2828)
and a few import fixes for free
2021-08-27 16:54:51 +00:00
Jacek Sieka 6a4bf98ea2
better error messages for keystore operations (#2812)
in particular, incluse os error string
2021-08-27 16:53:21 +00:00
Jacek Sieka a7a65bce42
disentangle eth2 types from the ssz library (#2785)
* reorganize ssz dependencies

This PR continues the work in
https://github.com/status-im/nimbus-eth2/pull/2646,
https://github.com/status-im/nimbus-eth2/pull/2779 as well as past
issues with serialization and type, to disentangle SSZ from eth2 and at
the same time simplify imports and exports with a structured approach.

The principal idea here is that when a library wants to introduce SSZ
support, they do so via 3 files:

* `ssz_codecs` which imports and reexports `codecs` - this covers the
basic byte conversions and ensures no overloads get lost
* `xxx_merkleization` imports and exports `merkleization` to specialize
and get access to `hash_tree_root` and friends
* `xxx_ssz_serialization` imports and exports `ssz_serialization` to
specialize ssz for a specific library

Those that need to interact with SSZ always import the `xxx_` versions
of the modules and never `ssz` itself so as to keep imports simple and
safe.

This is similar to how the REST / JSON-RPC serializers are structured in
that someone wanting to serialize spec types to REST-JSON will import
`eth2_rest_serialization` and nothing else.

* split up ssz into a core library that is independendent of eth2 types
* rename `bytes_reader` to `codec` to highlight that it contains coding
and decoding of bytes and native ssz types
* remove tricky List init overload that causes compile issues
* get rid of top-level ssz import
* reenable merkleization tests
* move some "standard" json serializers to spec
* remove `ValidatorIndex` serialization for now
* remove test_ssz_merkleization
* add tests for over/underlong byte sequences
* fix broken seq[byte] test - seq[byte] is not an SSZ type

There are a few things this PR doesn't solve:

* like #2646 this PR is weak on how to handle root and other
dontSerialize fields that "sometimes" should be computed - the same
problem appears in REST / JSON-RPC etc

* Fix a build problem on macOS

* Another way to fix the macOS builds

Co-authored-by: Zahary Karadjov <zahary@gmail.com>
2021-08-18 20:57:58 +02:00
Jacek Sieka 7a622e8505
rework spec imports (#2779)
The spec imports are a mess to work with, so this branch cleans them up
a bit to ensure that we avoid generic sandwitches and that importing
stuff generally becomes easier.

* reexport crypto/digest/presets because these are part of the public
symbol set of the rest of the spec types
* don't export `merge` types from `base` - this causes circular deps
* fix circular deps in `ssz/spec_types` - this is the first step in
disentangling ssz from spec
* be explicit about phase0 vs altair - longer term, `altair` will become
the "natural" type set, then merge and so on, so no point in giving
`phase0` special preferential treatment
2021-08-12 13:08:20 +00:00
Jacek Sieka 23eea197f6
Implement split preset/config support (#2710)
* Implement split preset/config support

This is the initial bulk refactor to introduce runtime config values in
a number of places, somewhat replacing the existing mechanism of loading
network metadata.

It still needs more work, this is the initial refactor that introduces
runtime configuration in some of the places that need it.

The PR changes the way presets and constants work, to match the spec. In
particular, a "preset" now refers to the compile-time configuration
while a "cfg" or "RuntimeConfig" is the dynamic part.

A single binary can support either mainnet or minimal, but not both.
Support for other presets has been removed completely (can be readded,
in case there's need).

There's a number of outstanding tasks:

* `SECONDS_PER_SLOT` still needs fixing
* loading custom runtime configs needs redoing
* checking constants against YAML file

* yeerongpilly support

`build/nimbus_beacon_node --network=yeerongpilly --discv5:no --log-level=DEBUG`

* load fork epoch from config

* fix fork digest sent in status
* nicer error string for request failures
* fix tools

* one more

* fixup

* fixup

* fixup

* use "standard" network definition folder in local testnet

Files are loaded from their standard locations, including genesis etc,
to conform to the format used in the `eth2-networks` repo.

* fix launch scripts, allow unknown config values

* fix base config of rest test

* cleanups

* bundle mainnet config using common loader
* fix spec links and names
* only include supported preset in binary

* drop yeerongpilly, add altair-devnet-0, support boot_enr.yaml
2021-07-12 15:01:38 +02:00