Commit Graph

396 Commits

Author SHA1 Message Date
Zahary Karadjov 0d6453aa51
Add an USE_VC env variable affecting the local simulation targets 2023-04-28 00:21:01 +03:00
tersec 228e10f1d9
update engine API URLs from v1.0.0-beta.2 to beta.3 (#4828) 2023-04-17 20:11:28 +00:00
Etan Kissling 450f06566b
accelerate execution layer sync using light client (#4805)
Turn on `--sync-light-client` option by default, now that it has shown
stability in local testnets.
2023-04-10 14:28:46 +00:00
tersec ed148ab69d
fix false positive getopt failure with multiple getopt matches in searched path (#4797)
* fix false positive getopt failure with multiple getopt matches in searched path

* also get launch_local_testnet

* also make_prometheus_config, called from launch_local_testnet
2023-04-08 00:18:29 +00:00
Zahary Karadjov 7f2a3b7130
[local sim] Download the latest nimbus-eth2 when --dl-nimbus-eth2 is used 2023-04-05 19:12:28 +03:00
Zahary Karadjov b4d731a1cc
Make it possible to run the local simulation on nixOS 2023-03-30 19:37:29 +03:00
zah 6539928775
Refer to binaries with their proper extensions on Windows (#4739) 2023-03-17 19:43:52 +01:00
Etan Kissling 2f006d31e7
rename `GETH_EIP_4844_BINARY` > `GETH_DENEB_BINARY` (#4729) 2023-03-12 01:47:24 +00:00
Etan Kissling 969c6f73ae
misc local `EIP4844` > `Deneb` bumps (#4717)
* misc local `EIP4844` > `Deneb` bumps

* fix
2023-03-11 00:28:19 +00:00
Zahary Karadjov 5370a6b67b
Capture the Geth output when running 'make zhejiang' 2023-03-10 11:13:24 +02:00
zah 8771e91d53
Support for driving multiple EL nodes from a single Nimbus BN (#4465)
* Support for driving multiple EL nodes from a single Nimbus BN

Full list of changes:

* Eth1Monitor has been renamed to ELManager to match its current
  responsibilities better.

* The ELManager is no longer optional in the code (it won't have
  a nil value under any circumstances).

* The support for subscribing for headers was removed as it only
  worked with WebSockets and contributed significant complexity
  while bringing only a very minor advantage.

* The `--web3-url` parameter has been deprecated in favor of a
  new `--el` parameter. The new parameter has a reasonable default
  value and supports specifying a different JWT for each connection.
  Each connection can also be configured with a different set of
  responsibilities (e.g. download deposits, validate blocks and/or
  produce blocks). On the command-line, these properties can be
  configured through URL properties stored in the #anchor part of
  the URL. In TOML files, they come with a very natural syntax
  (althrough the URL scheme is also supported).

* The previously scattered EL-related state and logic is now moved
  to `eth1_monitor.nim` (this module will be renamed to `el_manager.nim`
  in a follow-up commit). State is assigned properly either to the
  `ELManager` or the to individual `ELConnection` objects where
  appropriate.

  The ELManager executes all Engine API requests against all attached
  EL nodes, in parallel. It compares their results and if there is a
  disagreement regarding the validity of a certain payload, this is
  detected and the beacon node is protected from publishing a block
  with a potential execution layer consensus bug in it.

  The BN provides metrics per EL node for the number of successful or
  failed requests for each type Engine API requests. If an EL node
  goes offline and connectivity is resoted later, we report the
  problem and the remedy in edge-triggered fashion.

* More progress towards implementing Deneb block production in the VC
  and comparing the value of blocks produced by the EL and the builder
  API.

* Adds a Makefile target for the zhejiang testnet
2023-03-05 01:40:21 +00:00
Jakub Sokołowski 2f36f15b20
launch_local_testnet.sh: fix terminating Geth (#4684)
Also improve logging and list jobs after cleanup.

Signed-off-by: Jakub Sokołowski <jakub@status.im>
2023-03-01 18:27:55 +02:00
Jakub Sokołowski f46ed12f04
launch_local_testnet.sh: re-add targetting parent PID (#4680)
Otherwise we kill of other unrelated processes.

Fix for bug introduced most probably in:
https://github.com/status-im/nimbus-eth2/pull/4551

Signed-off-by: Jakub Sokołowski <jakub@status.im>
2023-02-28 10:39:39 +00:00
Jakub Sokołowski f2a09802b1
launch_local_testnet.sh: print processes first
Signed-off-by: Jakub Sokołowski <jakub@status.im>
2023-02-28 11:24:20 +01:00
Jakub Sokołowski 02437bb365
launch_local_testnet.sh: show which killed processes
First step in debugging issue most probably re-introduced by:
https://github.com/status-im/nimbus-eth2/pull/4551

Which causes the finalization tests script to kill other processes
unrelated to the given CI job.

Signed-off-by: Jakub Sokołowski <jakub@status.im>
2023-02-28 11:24:17 +01:00
Kim De Mey 14f4100e1b
Fix Nimbus EL node index in local testnet script (#4671) 2023-02-27 20:39:50 +01:00
zah 6036f2e7d7
Local sim impovements (#4551)
* Local sim impovements

* Added support for running Capella and EIP-4844 simulations
  by downloading the correct version of Geth.

* Added support for using Nimbus remote signer and Web3Signer.
  Use 2 out of 3 threshold signing configuration in the mainnet
  configuration and regular remote signing in the minimal one.

* The local testnet simulation can now use a payload builder.
  This is currently not activated in CI due to lack of automated
  procedures for installing third-party relays or builders.

  You are adviced to use mergemock for now, but for most realistic
  results, we can create a simple builder based on the nimbus-eth1
  codebase that will be able to propose transactions from the regular
  network mempool.

* Start the simulation from a merged state. This would allow us
  to start removing pre-merge functionality such as the gossip
  subsciption logic. The commit also removes the merge-forcing
  hack installed after the TTD removal.

* Consolidate all the tools used in the local simulation into a
  single `ncli_testnet` binary.
2023-02-23 02:10:07 +00:00
Jakub Sokołowski 3d3d17aad9
after_install: use . not source in bourne shell (#4639)
Another dumb mistake when using bourne shell:
```
/var/lib/dpkg/info/nimbus-beacon-node.postinst: 23: source: not found
```

Signed-off-by: Jakub Sokołowski <jakub@status.im>
2023-02-17 14:26:23 +01:00
Jakub Sokołowski 7d599185e0
use Bourne shell as the actual postinst script does (#4638)
The `postinst` wrapper script into which these scripts are embedded as
`after_upgrade` and `after_install` functions are executed using Bourne
shell(`sh`), so we cannot use the Bash specific `[[ ]]` test or it fails:
```
/var/lib/dpkg/info/nimbus-beacon-node.postinst: 22: [[: not found
```

Signed-off-by: Jakub Sokołowski <jakub@status.im>
2023-02-17 14:13:28 +02:00
Jakub Sokołowski 4aae1f0ee8
make DEB package descriptions long (#4620)
Apparently it's a good practice:
https://www.debian.org/doc/manuals/developers-reference/best-pkging-practices.html#the-long-description

Text mostly taken from https://nimbus.guide/.

Signed-off-by: Jakub Sokołowski <jakub@status.im>
2023-02-14 17:51:52 +01:00
Jakub Sokołowski b79d267b8f
drop dependency on lsb-release script (#4597)
The `/etc/os-release` file exists in most distributions and can be
easily read in Bash by sourcing it:
```
 > docker run --rm -it debian:bullseye
root@2f5d6e038738:/# grep '^ID=' /etc/os-release
ID=debian
```
```
 > docker run --rm -it ubuntu:22.04
root@316b572b6e4d:/# grep '^ID=' /etc/os-release
ID=ubuntu
```
The dependency on `lsb-release` tool
is unnecessary, and pulls in additional big dependencies like `python3`:
```
 # apt show lsb-release | grep Depends
Depends: python3:any, distro-info-data
```
Which if used in a Docker container would make it unnecessarily big.

Signed-off-by: Jakub Sokołowski <jakub@status.im>
2023-02-06 11:43:24 +01:00
Jakub Sokołowski c81352ce4e
ignore systemctl failure in after_install scripts (#4596)
Otherwise installation in Docker containers fails with:
```
...
Adding new user `nimbus' (UID 101) with group `nimbus' ...
Not creating home directory `/home/nimbus'.
/var/lib/dpkg/info/nimbus-beacon-node.postinst: 39: systemctl: not found
dpkg: error processing package nimbus-beacon-node (--configure):
 installed nimbus-beacon-node package post-installation script subprocess returned error exit status 127
Errors were encountered while processing:
 nimbus-beacon-node
E: Sub-process /usr/bin/dpkg returned an error code (1)
```

Signed-off-by: Jakub Sokołowski <jakub@status.im>
2023-02-06 11:01:57 +01:00
tersec bca781b1b0
prioritize REST API-provided BLS to execution changes over gossip-received changes (#4580) 2023-02-03 16:28:28 +01:00
tersec 1c62a5eb24
capella VC support (#4586) 2023-02-03 16:12:11 +01:00
tersec 63ed5885ab
update engine API URLs to v1.0.0-beta.2 (#4579) 2023-02-01 18:49:36 +00:00
tersec aacc8d702d
remove Nim 1.2-compatible `push raise`s and update copyright notice years (#4528) 2023-01-20 14:14:37 +00:00
zah 0f758c5f02
Working Makefile targets for Capella devnet2 (#4494)
* Working Makefile targets for Capella devnet2

make capella-devnet-2
make clean-capella-devnet-2

You'll need to have https://github.com/tmuxinator/tmuxinator installed.
It's available as a regular package in most Linux distributions or through
Nix or Brew on macOS.

This commit also fixes the initial hang in the Eth1 monitor in the "find
TTD block" procedure through a fix to the network metadata files which
hasn't been upstreamed yet.

Other changes:

* Disabled Geth snap sync in the simulation

When all Geth nodes are configured to run with snap sync enabled, they all
start snap sync after the first forkchoiceUpdated which causes the BNs to
skip validator duties because the EL is syncing. The snap sync never completes
due to poor connectivity between the Geth nodes in the simulation.
2023-01-13 12:21:58 +02:00
Zahary Karadjov 713bdd317d
Bugfix: Handle networks with TTD=0 (e.g. withdrawal devnets) 2023-01-10 19:17:11 +02:00
tersec b1acae67c3
revert to Geth stable commits for testnet (#4476) 2023-01-09 14:16:29 +00:00
Etan Kissling d80082c784
re-enable `--sync-light-client` in CI testnets (#4469)
libp2p issues related to operation cancellations have been addressed in
https://github.com/status-im/nim-libp2p/pull/816
This means we can once more enable `--sync-light-client` in CI, without
having to deal with spurious CI failures due to the cancellation issues.
2023-01-06 23:38:38 +01:00
Etan Kissling 44aa1f5152
avoid VC keymanager port conflict in CI (#4459)
`BASE_VC_KEYMANAGER_PORT` was not configured separately between runs,
make it configurable and base on `EXECUTOR_NUMBER` like the others.
2023-01-04 18:59:35 +01:00
tersec 121c66ad3e
fix getForkedBlock to read EIP4844 and Capella blocks (#4450)
* fix getForkedBlock to read EIP4844 and Capella blocks

* update geth develop URLs
2022-12-30 21:59:21 +00:00
tersec 2b7e2499d9
use unstable Geth version in CI for capella and eip4844 support (#4415) 2022-12-13 00:55:33 +00:00
zah d30cb8baf1
Support for obtaining deposit snapshots during trustedNodeSync (#4303)
Other changes:

* More optimal search for TTD block.

* Add timeouts to all REST requests during trusted node sync.
  Fixes #4037

* Removed support for storing a deposit snapshot in the network
  metadata.
2022-12-07 12:24:51 +02:00
Zahary Karadjov fbbab3bfef
Add version metric for the VC; Enable VC metrics in the local simulation 2022-11-30 12:47:11 +02:00
Jacek Sieka 04392c236b
bumps (#4352)
* build&style fixes
* use multithreading for LTO outside of make too
* blst 0.3.10 with no significant changes
2022-11-24 20:56:02 +00:00
zah d07113767d
Bugfix: The VC was producing invalid sync committee contributions (#4343)
Since the sync committee duties are no longer updated on every slot
and previously the sync committee aggregators selection proofs were
generated during the duties update, this now resulted in the client
using stale selection proofs (they must be generated at each slot).

The fix consists of moving the selection proof generation logic in
a different function which is properly executed on each slot.

Other changes:

* The logtrace tool has been enhanced with a framework for adding
  new simpler log aggregation and analysis algorithms.
  The default CI testnet simulation will now ensure that the blocks
  in the network have reasonable sync committee participation.
2022-11-24 09:46:35 +02:00
Etan Kissling ef2b434ec9
quicker Geth startup in local testnet (#4329)
Local testnet simulation currently waits 5 seconds when starting each
individual Geth instance. Waiting a shorter amount saves almost a minute
per minimum + mainnet CI finalization job.
Measured startup times per Geth: Linux ~100ms, macOS Intel ~300ms.
2022-11-17 11:40:27 +02:00
Etan Kissling bee3c4d68e
properly stop EL instances from prior testnets (#4328)
Launching multiple local testnet simulation sequentially can lead to
existing EL processes from prior failed/aborted runs not being stopped
properly, subsequently leading to hard-to-debug CI test failures.
Fixing the cleanup logic addresses this problem.
2022-11-16 12:12:20 +02:00
Jacek Sieka 1b90f65f7b
doc updates (#4314)
* optimise index / landing page real estate, add feature list
* fee recipient in quickstart
* recommend http over websockets  (fewer command line options needed)
* simplify PBS, fee recipient docs
* add VC role / sentry node docs (#4140)
* Update docs/the_nimbus_book/src/rest-api.md

Co-authored-by: tersec <tersec@users.noreply.github.com>
2022-11-13 07:59:55 +01:00
Etan Kissling 23cf8e9b22
temporarily disable LC sync in testnet (#4238)
LC sync seems to trigger libp2p disconnects in launch_local_testnet in
some cases. Disable the testing flag for now until investigated.
2022-10-14 15:12:14 +00:00
tersec 20d6481b39
increase local testnet validators from 128 to 1024 (#4214) 2022-10-04 19:44:20 +03:00
Jakub Sokołowski d697a54846
local_launch_testnet: use -SIGKILL instead of -9 (#4159)
Might help with this error:
```
pkill: illegal option -- 9
usage: pkill [-signal] [-ILfilnovx] [-F pidfile] [-G gid]
             [-P ppid] [-U uid] [-g pgrp] [-t tty] [-u euid]
             pattern ...
```

Signed-off-by: Jakub Sokołowski <jakub@status.im>

Signed-off-by: Jakub Sokołowski <jakub@status.im>
2022-09-22 11:55:50 +00:00
Etan Kissling 0708fcd7cf
rm require engine API check (#4144)
The `eth1_monitor` check to require engine API from bellatrix onward
has issues in setups where the EL and CL are started simultaneously
because the EL may not be ready to answer requests by the time that the
check is performed. This can be observed, e.g., on Raspberry Pi 4 when
using Besu as the EL client. Now that the merge transition happened, the
check is also not that useful anymore, as users have other ways to know
that their setup is not working correctly (e.g., repeated exchange logs)
2022-09-19 23:47:46 +02:00
tersec 5677816df3
remove obsolete merge test vector test bash script (#4135)
* remove obsolete merge test vector test bash script

* remove reference to removed merge test vector script
2022-09-16 22:49:39 +02:00
Mamy Ratsimbazafy f21495b9d4
Use engine API port in systemd.service fix #4098 (#4099) 2022-09-09 12:18:02 +00:00
Etan Kissling 317e91bfc1
support jwtsecret in `launch_local_testnet` (#4097)
To allow testing more thoroughly with connected EL, pass `jwtsecret`
as part of running local testnets.
2022-09-08 15:22:31 +02:00
Etan Kissling 613f4a9a50
accelerate EL sync with LC with `--sync-light-client` (#4041)
When the BN-embedded LC makes sync progress, pass the corresponding
execution block hash to the EL via `engine_forkchoiceUpdatedV1`.
This allows the EL to sync to wall slot while the chain DAG is behind.
Renamed `--light-client` to `--sync-light-client` for clarity, and
`--light-client-trusted-block-root` to `--trusted-block-root` for
consistency with `nimbus_light_client`.

Note that this does not work well in practice at this time:
- Geth sticks to the optimistic sync:
  "Ignoring payload while snap syncing" (when passing the LC head)
  "Forkchoice requested unknown head" (when updating to LC head)
- Nethermind syncs to LC head but does not report ancestors as VALID,
  so the main forward sync is still stuck in optimistic mode:
  "Pre-pivot block, ignored and returned Syncing"

To aid EL client teams in fixing those issues, having this available
as a hidden option is still useful.
2022-08-29 12:16:35 +00:00
tersec 61dc296046
update engine API spec ref URLs from alpha.9 to beta.1 (#4030)
* update engine API spec ref URLs from alpha.9 to beta.1

* require exactly 256-bit JWT keys
2022-08-26 13:44:50 +03:00
tersec 184c3de02d
remove MEV test script (#4031) 2022-08-26 10:02:07 +02:00