* Support for driving multiple EL nodes from a single Nimbus BN
Full list of changes:
* Eth1Monitor has been renamed to ELManager to match its current
responsibilities better.
* The ELManager is no longer optional in the code (it won't have
a nil value under any circumstances).
* The support for subscribing for headers was removed as it only
worked with WebSockets and contributed significant complexity
while bringing only a very minor advantage.
* The `--web3-url` parameter has been deprecated in favor of a
new `--el` parameter. The new parameter has a reasonable default
value and supports specifying a different JWT for each connection.
Each connection can also be configured with a different set of
responsibilities (e.g. download deposits, validate blocks and/or
produce blocks). On the command-line, these properties can be
configured through URL properties stored in the #anchor part of
the URL. In TOML files, they come with a very natural syntax
(althrough the URL scheme is also supported).
* The previously scattered EL-related state and logic is now moved
to `eth1_monitor.nim` (this module will be renamed to `el_manager.nim`
in a follow-up commit). State is assigned properly either to the
`ELManager` or the to individual `ELConnection` objects where
appropriate.
The ELManager executes all Engine API requests against all attached
EL nodes, in parallel. It compares their results and if there is a
disagreement regarding the validity of a certain payload, this is
detected and the beacon node is protected from publishing a block
with a potential execution layer consensus bug in it.
The BN provides metrics per EL node for the number of successful or
failed requests for each type Engine API requests. If an EL node
goes offline and connectivity is resoted later, we report the
problem and the remedy in edge-triggered fashion.
* More progress towards implementing Deneb block production in the VC
and comparing the value of blocks produced by the EL and the builder
API.
* Adds a Makefile target for the zhejiang testnet
First step in debugging issue most probably re-introduced by:
https://github.com/status-im/nimbus-eth2/pull/4551
Which causes the finalization tests script to kill other processes
unrelated to the given CI job.
Signed-off-by: Jakub Sokołowski <jakub@status.im>
* Local sim impovements
* Added support for running Capella and EIP-4844 simulations
by downloading the correct version of Geth.
* Added support for using Nimbus remote signer and Web3Signer.
Use 2 out of 3 threshold signing configuration in the mainnet
configuration and regular remote signing in the minimal one.
* The local testnet simulation can now use a payload builder.
This is currently not activated in CI due to lack of automated
procedures for installing third-party relays or builders.
You are adviced to use mergemock for now, but for most realistic
results, we can create a simple builder based on the nimbus-eth1
codebase that will be able to propose transactions from the regular
network mempool.
* Start the simulation from a merged state. This would allow us
to start removing pre-merge functionality such as the gossip
subsciption logic. The commit also removes the merge-forcing
hack installed after the TTD removal.
* Consolidate all the tools used in the local simulation into a
single `ncli_testnet` binary.
Another dumb mistake when using bourne shell:
```
/var/lib/dpkg/info/nimbus-beacon-node.postinst: 23: source: not found
```
Signed-off-by: Jakub Sokołowski <jakub@status.im>
The `postinst` wrapper script into which these scripts are embedded as
`after_upgrade` and `after_install` functions are executed using Bourne
shell(`sh`), so we cannot use the Bash specific `[[ ]]` test or it fails:
```
/var/lib/dpkg/info/nimbus-beacon-node.postinst: 22: [[: not found
```
Signed-off-by: Jakub Sokołowski <jakub@status.im>
The `/etc/os-release` file exists in most distributions and can be
easily read in Bash by sourcing it:
```
> docker run --rm -it debian:bullseye
root@2f5d6e038738:/# grep '^ID=' /etc/os-release
ID=debian
```
```
> docker run --rm -it ubuntu:22.04
root@316b572b6e4d:/# grep '^ID=' /etc/os-release
ID=ubuntu
```
The dependency on `lsb-release` tool
is unnecessary, and pulls in additional big dependencies like `python3`:
```
# apt show lsb-release | grep Depends
Depends: python3:any, distro-info-data
```
Which if used in a Docker container would make it unnecessarily big.
Signed-off-by: Jakub Sokołowski <jakub@status.im>
Otherwise installation in Docker containers fails with:
```
...
Adding new user `nimbus' (UID 101) with group `nimbus' ...
Not creating home directory `/home/nimbus'.
/var/lib/dpkg/info/nimbus-beacon-node.postinst: 39: systemctl: not found
dpkg: error processing package nimbus-beacon-node (--configure):
installed nimbus-beacon-node package post-installation script subprocess returned error exit status 127
Errors were encountered while processing:
nimbus-beacon-node
E: Sub-process /usr/bin/dpkg returned an error code (1)
```
Signed-off-by: Jakub Sokołowski <jakub@status.im>
* Working Makefile targets for Capella devnet2
make capella-devnet-2
make clean-capella-devnet-2
You'll need to have https://github.com/tmuxinator/tmuxinator installed.
It's available as a regular package in most Linux distributions or through
Nix or Brew on macOS.
This commit also fixes the initial hang in the Eth1 monitor in the "find
TTD block" procedure through a fix to the network metadata files which
hasn't been upstreamed yet.
Other changes:
* Disabled Geth snap sync in the simulation
When all Geth nodes are configured to run with snap sync enabled, they all
start snap sync after the first forkchoiceUpdated which causes the BNs to
skip validator duties because the EL is syncing. The snap sync never completes
due to poor connectivity between the Geth nodes in the simulation.
libp2p issues related to operation cancellations have been addressed in
https://github.com/status-im/nim-libp2p/pull/816
This means we can once more enable `--sync-light-client` in CI, without
having to deal with spurious CI failures due to the cancellation issues.
Other changes:
* More optimal search for TTD block.
* Add timeouts to all REST requests during trusted node sync.
Fixes#4037
* Removed support for storing a deposit snapshot in the network
metadata.
Since the sync committee duties are no longer updated on every slot
and previously the sync committee aggregators selection proofs were
generated during the duties update, this now resulted in the client
using stale selection proofs (they must be generated at each slot).
The fix consists of moving the selection proof generation logic in
a different function which is properly executed on each slot.
Other changes:
* The logtrace tool has been enhanced with a framework for adding
new simpler log aggregation and analysis algorithms.
The default CI testnet simulation will now ensure that the blocks
in the network have reasonable sync committee participation.
Local testnet simulation currently waits 5 seconds when starting each
individual Geth instance. Waiting a shorter amount saves almost a minute
per minimum + mainnet CI finalization job.
Measured startup times per Geth: Linux ~100ms, macOS Intel ~300ms.
Launching multiple local testnet simulation sequentially can lead to
existing EL processes from prior failed/aborted runs not being stopped
properly, subsequently leading to hard-to-debug CI test failures.
Fixing the cleanup logic addresses this problem.
The `eth1_monitor` check to require engine API from bellatrix onward
has issues in setups where the EL and CL are started simultaneously
because the EL may not be ready to answer requests by the time that the
check is performed. This can be observed, e.g., on Raspberry Pi 4 when
using Besu as the EL client. Now that the merge transition happened, the
check is also not that useful anymore, as users have other ways to know
that their setup is not working correctly (e.g., repeated exchange logs)
When the BN-embedded LC makes sync progress, pass the corresponding
execution block hash to the EL via `engine_forkchoiceUpdatedV1`.
This allows the EL to sync to wall slot while the chain DAG is behind.
Renamed `--light-client` to `--sync-light-client` for clarity, and
`--light-client-trusted-block-root` to `--trusted-block-root` for
consistency with `nimbus_light_client`.
Note that this does not work well in practice at this time:
- Geth sticks to the optimistic sync:
"Ignoring payload while snap syncing" (when passing the LC head)
"Forkchoice requested unknown head" (when updating to LC head)
- Nethermind syncs to LC head but does not report ancestors as VALID,
so the main forward sync is still stuck in optimistic mode:
"Pre-pivot block, ignored and returned Syncing"
To aid EL client teams in fixing those issues, having this available
as a hidden option is still useful.
* Fixes a segfault during block production when the Keymanager API
is disabled. The Keymanager is now disabled on half of the local
testnet nodes to catch such problems in the future.
* Fixes multiple potential stalls from REST requests being done
without a timeout. From practice, we know that such requests
can hang forever if not cancelled with a timeout. At best,
this would be a resource leak, at worst, it may lead to a
full stall of the client and missed validator duties.
* Changes some Options usages to Opt (for easier use of valueOr)
* Keymanager API for the validator client
* Properly treat the 'description' field as optional when loading Keystores
* Spec-compliant serialization of the slashing data in Keymanager's DeleteKeys response ()
Fixes#3940Fixes#3964Closes#3884 by adding test
* packaging updates
* one package per binary (nimbus_beacon_node, nimbus_validator_client)
* use `-` in package name (`_` is separating the version)
* don't include (un)installation scripts in package
* default metrics port 8108 for vc
* fix several upgrade/install errors in scripts
* add JWT option to service files
* don't attempt to remove user on purge
* Use final `v1` version for light client protocols
* Unhide LC data collection options
* Default enable LC data serving
* rm unneeded import
* Connect to EL on startup
* Add docs for LC based EL sync
The ports for the concurrently executing REST and Minimal testnet clash,
leading to some CI failures since #3827 introduced further concurrency.
Adjusting the ports to be distinct across various tests should fix this.
Adds the `--web3-url` launch argument to `nimbus_light_client` to enable
driving the EL with the optimistic head obtained from LC sync protocol.
This will keep issuing `newPayload` / `forkChoiceUpdated` requests for
new blocks, marking them as optimistic. `ZERO_HASH` is reported as the
finalized block for now.
`a48d741022f6b0da1bb679e0ede4e38c019242cf` disabled LC in local testnet
as an undocumented side effect. Re-enabling for more thorough testing,
and added handling of LC with `--eth2-docker-image`.
* remove web3 url prompt in launcher script
The interactive prompt for web3 has outlived its utility as we now load
url:s from command line params and config files, preventing the prompt
from correctly detecting when it's needed.
Also, after the merge, a JWT secret will (likely) be needed.
* log notice when web3 url is missing
* fix docs to not mention default that doesn't exist
* fix scripts to properly quote arguments
* Add support for launching Geth and Nimbus-eth1 nodes in launch_local_simulation
* Prevent undefined variables errors when Nimbus and Geth are not launched
Co-authored-by: Nikolay Mitev <github@hmel.org>
* systemd: use same service file as package
* systemd: recommend setting up a user
* clean up obsolete / ephemeral notes
* eth1 -> execution, in many places
* clean up docker instructions to match the experience of running the
runner script
* add basic instructions / links for execution clients other than geth
* clear a few more references to JSON-RPC, update to REST
* update hardware requirements with execution client extras
* start talking about merge requirements (this needs a dedicated page)
* note that binaries do exist, where relevant
* list gitcoin under donations
* update syncing docs
* `NOT` is for `NOTICE` in logs
* update rpi hardware recommendations
* prevent automatic doppelganger restarts in systemd service file
For consistency with other options, use a common prefix for light client
data configuration options.
* `--serve-light-client-data` --> `--light-client-data-serve`
* `--import-light-client-data` --> `--light-client-data-import-mode`
No deprecation of the old identifiers as they were only sparingly used
and all usage can be easily updated without interferance.
When querying `ALTAIR_FORK_EPOCH` fails, the `launch_local_testnet`
script got stuck because comparing `CURRENT_FORK_EPOCH` against it
would keep failing with `[: : integer expression expected`.
Querying `ALTAIR_FORK_EPOCH` is now retried until a number is returned.
When launched with `--light-client-enable` the latest blocks are fetched
and optimistic candidate blocks are passed to a callback (log for now).
This helps accelerate syncing in the future (optimistic sync).
Introduces a new library for syncing using libp2p based light client
sync protocol, and adds a new `nimbus_light_client` executable that uses
this library for syncing. The new executable emits log messages when
new beacon block headers are received, and is integrated into testing.
Other changes:
* logtrace can now verify sync committee messages and contributions
* Many unnecessary use of pairs() have been removed for consistency
* Map 40x BN response codes to BeaconNodeStatus.Incompatible in the VC
Other fixes:
* Fix bit rot in the `make prater-dev-deposit` target.
* Correct content-type in the responses of the Nimbus signing node
* Invalid JSON payload was being sent in the web3signer requests
* Jenkins: get rid of macOS warnings
We're linking object files with debug info with ones without it. Perfectly justified, from our point of view, but dsymutil complains verbosely about it, with no switch to silence it.
Zahary Karadjov
Jakub Sokołowski
Kim De Mey
tersec
Etan Kissling
Jacek Sieka
Mamy Ratsimbazafy
Nikolay Mitev
Miran
Zahary Karadjov
Ștefan Talpalaru