Commit Graph

185 Commits

Author SHA1 Message Date
tersec ed148ab69d
fix false positive getopt failure with multiple getopt matches in searched path (#4797)
* fix false positive getopt failure with multiple getopt matches in searched path

* also get launch_local_testnet

* also make_prometheus_config, called from launch_local_testnet
2023-04-08 00:18:29 +00:00
zah 8771e91d53
Support for driving multiple EL nodes from a single Nimbus BN (#4465)
* Support for driving multiple EL nodes from a single Nimbus BN

Full list of changes:

* Eth1Monitor has been renamed to ELManager to match its current
  responsibilities better.

* The ELManager is no longer optional in the code (it won't have
  a nil value under any circumstances).

* The support for subscribing for headers was removed as it only
  worked with WebSockets and contributed significant complexity
  while bringing only a very minor advantage.

* The `--web3-url` parameter has been deprecated in favor of a
  new `--el` parameter. The new parameter has a reasonable default
  value and supports specifying a different JWT for each connection.
  Each connection can also be configured with a different set of
  responsibilities (e.g. download deposits, validate blocks and/or
  produce blocks). On the command-line, these properties can be
  configured through URL properties stored in the #anchor part of
  the URL. In TOML files, they come with a very natural syntax
  (althrough the URL scheme is also supported).

* The previously scattered EL-related state and logic is now moved
  to `eth1_monitor.nim` (this module will be renamed to `el_manager.nim`
  in a follow-up commit). State is assigned properly either to the
  `ELManager` or the to individual `ELConnection` objects where
  appropriate.

  The ELManager executes all Engine API requests against all attached
  EL nodes, in parallel. It compares their results and if there is a
  disagreement regarding the validity of a certain payload, this is
  detected and the beacon node is protected from publishing a block
  with a potential execution layer consensus bug in it.

  The BN provides metrics per EL node for the number of successful or
  failed requests for each type Engine API requests. If an EL node
  goes offline and connectivity is resoted later, we report the
  problem and the remedy in edge-triggered fashion.

* More progress towards implementing Deneb block production in the VC
  and comparing the value of blocks produced by the EL and the builder
  API.

* Adds a Makefile target for the zhejiang testnet
2023-03-05 01:40:21 +00:00
zah 6036f2e7d7
Local sim impovements (#4551)
* Local sim impovements

* Added support for running Capella and EIP-4844 simulations
  by downloading the correct version of Geth.

* Added support for using Nimbus remote signer and Web3Signer.
  Use 2 out of 3 threshold signing configuration in the mainnet
  configuration and regular remote signing in the minimal one.

* The local testnet simulation can now use a payload builder.
  This is currently not activated in CI due to lack of automated
  procedures for installing third-party relays or builders.

  You are adviced to use mergemock for now, but for most realistic
  results, we can create a simple builder based on the nimbus-eth1
  codebase that will be able to propose transactions from the regular
  network mempool.

* Start the simulation from a merged state. This would allow us
  to start removing pre-merge functionality such as the gossip
  subsciption logic. The commit also removes the merge-forcing
  hack installed after the TTD removal.

* Consolidate all the tools used in the local simulation into a
  single `ncli_testnet` binary.
2023-02-23 02:10:07 +00:00
zah 067ba13c52
Capella metadata for Sepolia (#4615)
Other changes:

Renamed the `EIP_4844_FORK_*` config constants to `DENEB_FORK_*` as
this matches the latest spec and it's already used in the official
Sepolia config.
2023-02-15 14:44:09 +00:00
Jacek Sieka 02b48fafad
remove unused / obsolete / unmaintained modules (#4298)
mostly this is chronicles-tail and its (extensive) dependencies along
with the simulation monitoring dashboard that is not maintained
2022-11-08 14:37:23 +01:00
tersec 5b0b48f6e9
implement /eth/v1/validator/register_validator (#4115) 2022-09-13 14:52:26 +03:00
tersec 92c1e75915
stop using --rpc in run-beacon-node and simulations (#3662) 2022-05-24 16:21:55 +00:00
Jacek Sieka bc80ac3be1
harden REST API `atSlot` against non-finalized blocks (#3538)
* harden validator API against pre-finalized slot requests
* check `syncHorizon` when responding to validator api requests too far
from `head`
* limit state-id based requests to one epoch ahead of `head`
* put historic data bounds on block/attestation/etc validator production API, preventing them from being used with already-finalized slots
* add validator block smoke tests
* make rest test create a new genesis with the tests running roughly in
the first epoch to allow testing a few more boundary conditions
2022-03-23 12:42:16 +01:00
Ștefan Talpalaru d5a2c75963
restapi.sh: cleanup on exit (#3328)
also rename a confusing option/var combo
2022-01-27 13:03:38 +01:00
Zahary Karadjov 54d0d588b1 Implementation of the Keymanager API (BETA)
https://github.com/ethereum/keymanager-APIs
2022-01-04 18:51:45 +02:00
Zahary Karadjov 6b4f32ae23
Replicate a recent fix from the launch_local_testnet script due to a wide-spread code duplication 2021-12-22 17:59:45 +02:00
cheatfate b3ee5d67bd Remove nimbus_signing_process. 2021-11-30 16:48:36 +02:00
Zahary Karadjov 5e62df5ad5 /bin/bash -> /usr/bin/env bash (needed on NixOS) 2021-11-15 15:45:01 +02:00
Jacek Sieka 233d756518
Logging and startup improvements (#3038)
* Logging and startup improvements

Color support for released binaries!

* startup scripts no longer log to file by default - this only affects
source builds - released binaries don't support file logging
* add --log-stdout option to control logging to stdout (colors, json)
* detect tty:s vs redirected logs and log accordingly
* add option to disable log colors at runtime
* simplify several "common" logs, showing the most important information
earlier and more clearly
* remove line numbers / file information / tid - these take up space and
are of little use to end users
  * still enabled in debug builds and tools
* remove `testnet_servers_image` compile-time option
* server images, released binaries and compile-from-source now offer
the same behaviour and features
* fixes https://github.com/status-im/nimbus-eth2/issues/2326
* fixes https://github.com/status-im/nimbus-eth2/issues/1794
* remove instanteneous block speed from sync message, keeping only
average

before:

```
INF 2021-10-28 16:45:59.000+02:00 Slot start                                 topics="beacnde" tid=386429 file=nimbus_beacon_node.nim:884 lastSlot=2384027 wallSlot=2384028 delay=461us84ns peers=0 head=75a10ee5:3348 headEpoch=104 finalized=cd6804ba:3264 finalizedEpoch=102 sync="wwwwwwwwww:0:0.0000:0.0000:00h00m (3348)"
INF 2021-10-28 16:45:59.046+02:00 Slot end                                   topics="beacnde" tid=386429 file=nimbus_beacon_node.nim:821 slot=2384028 nextSlot=2384029 head=75a10ee5:3348 headEpoch=104 finalizedHead=cd6804ba:3264 finalizedEpoch=102 nextAttestationSlot=-1 nextProposalSlot=-1 nextActionWait=n/a
```

after:

```
INF 2021-10-28 22:43:23.033+02:00 Slot start                                 topics="beacnde" slot=2385815 epoch=74556 sync="DDPDDPUDDD:10:5.2258:01h19m (2361088)" peers=37 head=eacd2dae:2361096 finalized=73782:a4751487 delay=33ms687us715ns
INF 2021-10-28 22:43:23.291+02:00 Slot end                                   topics="beacnde" slot=2385815 nextActionWait=n/a nextAttestationSlot=-1 nextProposalSlot=-1 head=eacd2dae:2361096
```

* fix comment

* documentation updates

* mention `--log-file` may be deprecated in the future
* update various docs
2021-11-02 18:06:36 +01:00
Etan Kissling 6720e6e2a6 fix running local validator client simulation
Updated the local network simulation scripts to use the REST interface
for connecting the validator client to the beacon node. Otherwise, the
current scripts produce errors.
2021-10-21 00:47:33 +03:00
zah 3689c68cbf
Carry out the sync committee gossip duties
Other changes:

* Add server getBlockV2(), and produceBlockV2().
* Add getBlockV2() to REST test suite.
* Add client getBlockV2(), and produceBlockV2().
* Fix URLs in comments.
* Add some primitives and fix some issues in forks.nim.
* Switch `validator_client` to V2 calls usage.
* Bump `chronos` with imports fixes.
* Bump `nim-json-serialization` for `requireAllFields`.
2021-08-30 03:58:30 +03:00
Ștefan Talpalaru 59b1a4772c
don't try to use lsof on Windows (#2762) 2021-08-05 10:49:34 +02:00
Ștefan Talpalaru 17c94c4d8e
restapi.sh: prevent port clashes (#2746) 2021-07-29 18:34:01 +02:00
Eugene Kabanov 4ba69bf54d
Disable discovery5 service for REST test. (#2738)
Run REST test before Finalization test.
2021-07-26 19:55:24 +00:00
zah d9f2a91374
Remove the obsolete testnet0/1 scripts (#2727)
Also fixes `make eth2_network_simulation`
2021-07-16 13:02:27 +03:00
Jacek Sieka 23eea197f6
Implement split preset/config support (#2710)
* Implement split preset/config support

This is the initial bulk refactor to introduce runtime config values in
a number of places, somewhat replacing the existing mechanism of loading
network metadata.

It still needs more work, this is the initial refactor that introduces
runtime configuration in some of the places that need it.

The PR changes the way presets and constants work, to match the spec. In
particular, a "preset" now refers to the compile-time configuration
while a "cfg" or "RuntimeConfig" is the dynamic part.

A single binary can support either mainnet or minimal, but not both.
Support for other presets has been removed completely (can be readded,
in case there's need).

There's a number of outstanding tasks:

* `SECONDS_PER_SLOT` still needs fixing
* loading custom runtime configs needs redoing
* checking constants against YAML file

* yeerongpilly support

`build/nimbus_beacon_node --network=yeerongpilly --discv5:no --log-level=DEBUG`

* load fork epoch from config

* fix fork digest sent in status
* nicer error string for request failures
* fix tools

* one more

* fixup

* fixup

* fixup

* use "standard" network definition folder in local testnet

Files are loaded from their standard locations, including genesis etc,
to conform to the format used in the `eth2-networks` repo.

* fix launch scripts, allow unknown config values

* fix base config of rest test

* cleanups

* bundle mainnet config using common loader
* fix spec links and names
* only include supported preset in binary

* drop yeerongpilly, add altair-devnet-0, support boot_enr.yaml
2021-07-12 15:01:38 +02:00
Eugene Kabanov 41c2d5f040
Store REST test logs as CI artifacts. (#2699)
* Store resttest logs as artifacts.

* Specify data-dir and sleep-timeout.
2021-07-06 08:40:18 +00:00
Eugene Kabanov 32702d37a3
Move REST simulation test to Jenkins. (#2688) 2021-06-30 08:57:07 +00:00
Zahary Karadjov ed2f6f753d Allow the custom testnet metadata to specify a path to the genesis file 2021-06-01 15:50:50 +03:00
Eugene Kabanov 0c3302b826
REST API test framework and tests. (#2492)
* REST API test framework and tests.

* Fix ValidatorIndex tests to properly handle int32, but not uint32 values.

* Fix tests to follow latest REST fixes.

* refactor restapi.sh

and add it to the test suite

* Fix issues.
Add delay timeout which is required.

* Fix restapi.sh script for Windows.

Co-authored-by: Ștefan Talpalaru <stefantalpalaru@yahoo.com>
2021-04-27 23:46:24 +03:00
Jacek Sieka f821bc878e
Remove `-d:insecure` compile option (#2468)
With metrics running on top of chronos, the metrics server no longer
needs to be compiled in conditionally - it remains disabled by default.
2021-04-01 14:44:11 +02:00
Dustin Brody 67e4a045a3 simplify doppelganger detection to boolean 2021-02-03 20:55:33 +02:00
tersec 6141286547
rename doppelganger protection to doppelganger detection; switch default from warn to stop (#2281) 2021-02-01 12:18:16 +01:00
Dustin Brody 281853eee8 rename options and internal structures to doppelgangerFoo and remove probing 2021-01-30 00:17:54 +02:00
Dustin Brody a16f5afcd5 pre-emptive duplicate validator detection heuristic 2021-01-21 16:03:02 +02:00
Zahary Karadjov e9b9cd75ee Rename binaries; Mimic the original repo layout in the distribution 2020-11-09 11:38:52 +02:00
Eugene Kabanov eee01a32d5
Regression fix of eth2_network_simulation on Windows. (#1900)
* Concentrate all sensitive writeFile/createPath calls in one place.
Fix eth2_network_simulation for Windows.

* Remove artifacts.

* fix import

Co-authored-by: Jacek Sieka <jacek@status.im>
2020-10-27 12:04:17 +01:00
Zahary Karadjov 2ea5385aa0 Fix invalid file permissions causilg make eth2_network_sim to fail 2020-10-07 18:12:21 +03:00
Zahary Karadjov aed291128a Add support for starting from weak subjectivity checkpoints
Also removes the `genesis.ssz` file stored in the data folder.
The `medalla-fast-sync` target has been adapted to use the new features.
2020-10-07 09:32:03 +03:00
cheatfate add22a20e1 Update local_testnet and simulation scripts to use netkey-file and insecure-netkey-password.
Add more logging
2020-10-05 22:19:50 +03:00
Viktor Kirilov d9f9949ef0 use a separate process for the private keys (Off by default) - there is a new signing_process binary which loads all validators of the beacon node and the BN dictates through stdin of the signing process what to be signed and when and reads from stdout of the process 2020-09-02 12:47:00 +03:00
Viktor Kirilov 65d7787b1e 50/50 bn/vc split for the validator keys ON by default for the testnet scripts 2020-09-01 16:39:07 +03:00
tersec af3355e0f8
create local testnet mode for eth2_network (#1494) 2020-08-12 14:16:59 +00:00
Viktor Kirilov 5bbeb38f2d fixes the BN/VC communication - properly getting the attestation duties & also fixed start.sh 2020-08-06 15:29:05 +03:00
protolambda e90c5440e8 make eth1 distance runtime configurable 2020-08-06 14:49:58 +03:00
Zahary Karadjov 4b8ebb5d71 Correct instructions in the README for running prometheus on Medalla 2020-08-05 19:28:35 +03:00
Ștefan Talpalaru fa9f35e148
Jenkins: run local testnet test on macOS (#1391) 2020-07-29 14:08:27 +02:00
Zahary Karadjov f4c19e303a Non-interactive generation of keystores in the local sim 2020-07-28 07:36:25 +03:00
Zahary Karadjov 40ea9e9886 Integrate the Wallet support with the Deposits creation; Produce Launchpad-compatible deposits 2020-07-28 07:36:25 +03:00
Ștefan Talpalaru e9193fc9da
eth2_network_simulation: fixes (#1368)
- delete "tests/simulation/{data,validators}" by default, because old
  validator keys can lead to a crash with a cryptic error message
- actually start the Prometheus daemon on `make eth2_network_simulation`
- kill any running Prometheus daemon on exit
- fix some shell script syntax incompatible with Bash
2020-07-23 17:58:54 +02:00
Viktor Kirilov 4767eba465 - properly distributing the keys between BNs & VCs - fixing finalization - will switch to ON by default as soon as finalization becomes overall stable
- added a new altona-vc Makefile target which uses a separate VC process
2020-07-15 16:18:21 +03:00
Zahary Karadjov 540b2828b2
Adapt the local sim scripts to use the new run-time presets 2020-07-10 01:08:54 +03:00
Zahary Karadjov 302cef5e90 Attach all local validators before starting the networking 2020-06-29 18:07:41 +03:00
Zahary Karadjov 8140b4458c Working local sim with WAIT_GENESIS=yes 2020-06-29 02:18:48 +03:00
Zahary Karadjov 9cd8a6c730 Working local sim (without waiting genesis) 2020-06-29 02:18:48 +03:00