Nimbus `v22.9.0` is a `high-urgency` upgrade that fixes a critical pre-TTD block production issue affecting users that restarted their node after Bellatrix. It also improves compatibility with Besu, Prysm and slow block builders and provides a speed boost in block processing important for those running on Raspberry Pi and similar hardware.
With the merge drawing near, the focus of this release has been to include low risk changes that improve stability and compatibility - if you are unsure whether to upgrade, do reach out to us in discord to discuss your particular deployment.
A shout out to our great community for reporting and helping diagnose the issues that led up to this release - in particular Michael Sproul (Lighthouse) and Joe Clapis (Rocket Pool).
### Improvements
* Allow more time for block builder to deliver block
Nimbus `v22.8.2` is a `low-urgency` hotfix release, eliminating a risk for potential crash during block production that was introduced in the `v22.8.1` release. You can safely skip this release if you haven't enabled DEBUG logging on your beacon node, as the risk exists only when DEBUG logging is enabled.
Nimbus `v22.8.1` is a `high-urgency` upgrade, improving the stability and performance of Nimbus in post-merge networks. Upgrading is highly recommended due to improved timing of the interactions with the execution engine which may lead to higher profitability from block production, especially for users running Nethermind.
### Improvements
* More timely block proposals in the presence of a non-responsive builder node:
Nimbus `v22.8.0` is a `medium` urgency release, featuring full support for the upcoming mainnet merge! All users should upgrade at their earliest convenient, but no later than 5th of September.
> Since the network will go through the Bellatrix hard-fork on Sept 6, 2022, 11:34:47am UTC, failure to upgrade in time will result in inactivity penalties.
Please note that once the network reaches the terminal total difficulty (currently estimated to happen between 13th and 15th of September), it will no longer be possible to operate a beacon node without pairing it with a single non-shared merge-ready execution client. Nimbus is fully compatible will all execution clients and the required configuration steps for all of them are the same. Please refer to our merge guide for more details:
https://nimbus.guide/merge.html
To raise awareness of the required configuration changes, once the Bellatrix fork is activated on 6th of September, Nimbus will refuse to start unless a properly configured and authenticated Engine API end-point is provided through the command-line options `--web3-url` and `--jwt-secret`. If you need more time to complete the transition, you can temporarily run the beacon node with the command-line option `--require-engine-api-in-bellatrix=no`, but please note that such a setup will stop working once the network TTD is reached!
We would like to say a huge THANK YOU to all of our users who provided immensely valuable feedback in the many months of hard work leading to the merge and to all the fellow research and implementation teams who made this historic release possible!
Onwards and happy merging!
### Breaking changes
* Nimbus will refuse to start unless connected to a properly configured execution client in Bellatrix-enabled networks:
* The custom error code returned by Nimbus when a validator doppelganger is detected has been changed from 1031 to 129 to improve compatibility with `systemd`:
* Beta release for the Nimbus stand-alone light client, which can be used to drive any execution client without requiring a full-blown beacon node:
https://nimbus.guide/light-client-data.html
* The first spec-compliant implementation of the LibP2P protocols for serving light client data:
https://nimbus.guide/light-client-data.html
* Keystore locking prevents accidentally loading the same validator keys in multiple instances of the Nimbus beacon node and the Nimbus validator client, thus eliminating a potential slashing risk:
* Debian and RPM packages for the Nimbus beacon node and the Nimbus validator client are now available as part of the release. In the near future, Status will also provide a package repository, offering a more convenient installation:
* The Nimbus validator client now supports validator activity metrics such as `beacon_attestations_sent`, `beacon_aggregates_sent`, `beacon_attestation_sent_delay`, `beacon_blocks_sent`, `beacon_blocks_sent_delay`, `beacon_sync_committee_messages_sent`, `beacon_sync_committee_message_sent_delay`, `beacon_sync_committee_contributions_sent`:
* The sync status displayed in the Nimbus status bar and certain log messages now describes the state of the client more accurately (optimistically synced vs fully synced):
Nimbus `v22.7.0` is a `low` urgency release packing everything necessary for the [upcoming Prater/Goerli merge](https://wenmerge.com/) and introducing the [Nimbus validator client](https://nimbus.guide/validator-client.html) (currently in BETA).
### Other Improvements
* Support for fee recipient management through the [Keymanager API](https://ethereum.github.io/keymanager-APIs/#/Fee%20Recipient) and
through the [`/eth/v1/validator/prepare_beacon_proposer`](https://ethereum.github.io/beacon-APIs/#/Validator/prepareBeaconProposer) Beacon API end-point:
Nimbus `v22.6.1` is a `low-urgency` release which comes pre-configured with the correct TTD value for the Sepolia network and improves the behavior of Nimbus in merge testnets.
Nimbus `v22.6.0` brings support for the merge testnets Ropsten and Sepolia (please stay tuned for TTD announcements for the latter) and a lot of polish where we've taken the time to address a long list of UX improvements and bug fixes suggested or reported by our users. We are deeply grateful to everybody who contributed valuable feedback for this release.
* The custom error code `129` will signal a detected doppelganger on the network. This can be handled in the Nimbus's service supervisor to prevent an automatic restart:
* Incorrect encoding of the `current_epoch_participation` and `previous_epoch_participation` fields in the REST requests returning `BeaconState` results:
* The Nimbus-specific JSON-RPC service which was deprecated in version v22.3.0 is now removed. If you are currently relying on the JSON-RPC API, please consider switching to the official [REST API](https://nimbus.guide/rest-api.html). Using any of the `--rpc` flags will now result in a warning:
Nimbus `v22.5.0` is a `low-urgency` maintenance release. It implements the proposer boosting fork-choice policy and is compliant with the latest [Bellatrix specifications](https://github.com/ethereum/consensus-specs#bellatrix-also-known-as-the-merge). It also provides an early preview of our built-in support for [BLS threshold signatures](https://notes.ethereum.org/@djrtwo/blst-rfp) (via regular Web3Signer instances): this marks the first step of our long-term [secret-shared validators roadmap](https://github.com/status-im/nimbus-eth2/issues/3416) which enables node operators / staking pools to deploy Nimbus in secure high availability setups (guaranteeing ~100% uptime).
* An initial preview release fеaturing built-in support for distributed keystores, (part of our [secret shared validators roadmap]( https://github.com/status-im/nimbus-eth2/issues/3416)):
Nimbus `v22.4.0` is a `low-urgency` upgrade which brings with it further optimisations, and better user experience around [trusted node sync](https://nimbus.guide/trusted-node-sync.html). It lays the foundations for upcoming the merge hard-fork which will be fully supported in our next release (`v22.5.0`).
* The REST API now *only* returns current and relevant information in response to VC queries: in other words, information from the recent non-finalized portion of the chain history:
* Better and more consistent gossip mesh health: the `--max-peers` option now works as a target that can be exceeded by the client temporarily in order to maintain good gossip mesh health; the newly introduced `--hard-max-peers` option now acts as the hard limit that should not be exceeded (default set to `max-peers * 1.5`):
* An [ERA files](https://our.status.im/nimbus-update-march/#era-files-a-proposed-solution-to-historical-data-queries) developer preview: ERA files are an ultra-efficient long-term storage format for finalized chain history:
Nimbus `v22.3.0` is a `low-urgency` upgrade that marks the beginning of a more predictable release cadence for Nimbus. Going forward, we'll be publishing a new release each month, following a feature freeze period with intensified testing and monitoring of the introduced code changes on our dispersed fleet of mainnet validators.
> Please note that the new versioning scheme is tied to the calendar. The number 22 indicates the year of the release (2022), while 3 is the month (March). The last digit is the patch number of the release and it will have a non-zero value only when we ship a hotfix during the month.
* Extremely slow [slashing DB import](https://nimbus.guide/migration.html#step-4---import-your-slashing-protection-history) for validators with long validation history: the import should be nearly instant now
- The [JSON-RPC](https://nimbus.guide/api.html) service (`--rpc` flag) option is now deprecated. It's scheduled for removal in version `v22.6` (i.e. June of this year). If you are currently relying on the JSON-RPC API, please consider switching to the official [REST API](https://nimbus.guide/rest-api.html).
Nimbus `v1.7.0` is a `low-urgency` feature-packed upgrade, which brings support for [trusted node sync](https://nimbus.guide/trusted-node-sync.html) (also known as checkpoint sync) and HTTPS web3 providers.
Of particular note: the [Keymanager API](https://nimbus.guide/keymanager-api.html) now supports remote keystores (a.k.a web3signer keystores).
- Nimbus will no longer rewrite HTTP(S) web3 URLs to their respective WebSocket alternatives. Please review your setup to ensure you are using the desired web3 end-point.
- The peer scoring has been further tuned. As such the `--max-peers` should not be set below 70. Note that Lowering `max-peers` does not significantly improve bandwidth usage, but does increase the risk of missed attestations.
* Nimbus now treats the first `--web3-url` as a primary and preferred web3 provider. Any extra URLs are treated as fallback providers (to be used only when the primary is offline). As soon as the primary is usable again, Nimbus will switch back to it.
* The Keymanager API now supports management of remote keystores (also known as web3signer keystores):
* 128MB of savings come from exploiting a provision in the official spec, which allows clients to respond with only non-finalized blocks to network queries which request blocks by their root hash.
* A new `--rest-url` parameter for the `deposits exit` command: https://github.com/status-im/nimbus-eth2/pull/3344, https://github.com/status-im/nimbus-eth2/pull/3318
* You can now issue exits uing any beacon node which provides the [official REST API](https://nimbus.guide/rest-api.html). The Nimbus-specific [JSON-RPC API](https://nimbus.guide/api.html) will be deprecated in our next release, with a view to completely phasing it out over the next few months.
* Unnecessary CPU and bandwidth usage: https://github.com/status-im/nimbus-eth2/pull/3308
* The result of staying subsribed to sync committee topics even when there were no validators in the committee.
* Excessive logging on beacon nodes with large numbers of validators (in particular, those with `--validator-monitor-totals` enabled): https://github.com/status-im/nimbus-eth2/pull/3332
* Deviations from the spec in the REST API; this led to sub-optimal performance when Nimbus was paired with Vouch.
* Naming inconsistencies in the "totals" metrics (this was produced by the [validator monitor](https://nimbus.guide/validator-monitor.html)).
* Non-compliant implementation of the `/eth/v1/node/health` API (we were not producing HTTP status codes as mandated by the spec).
* Unnecessary restarts of the Eth1 syncing progress when the web3 provider connection was lost during sync: https://github.com/status-im/nimbus-eth2/pull/3354
* Tuned peer management: reduces the likelihood of missed attestations
* If you've seen frequent "No peers for topic" in your logs, this release will help
* Improved buffer management in the networking layer: reduces both CPU and memory usage.
* Further optimised batch verification of gossip messages: provides a 2-fold improvement in throughput.
* Comprehensive set of metrics for live validator performance monitoring in Grafana and support for producing detailed historic reward analysis in `ncli_db` (note that `ncli_db` is available only when compiling from source at the moment).
* Support for the new Keymanager API: add, remove, and migrate validators on the fly (BETA).
* Blazingly fast historical traversals in the REST API for beacon chain data mining: state caching brings up to a 10x speed-up in some common usage patterns (e.g. obtaining historic data slot by slot or epoch by epoch).
* 3x speed-up in snappy compression and decompression.
* Support for obtaining JSON payloads from the REST API.
Nimbus `v1.5.5` is a `medium-urgency` bugfix release which contains a number of significant optimisations; of particular note is a **6x speed-up in epoch processing** and **2x speed up in Altair block processing**.
In addition, `v1.5.5` adds support for the `web3signer` protocol (currently in BETA).
### We've fixed:
* The potential for missed block proposals when a third-party validator client (with at least one imported validator) is used with a Nimbus beacon node (with no imported validators)
* The web3 connection not being enabled when running third-party validator clients
* A rare condition in which the REST service becomes unavailable.
* Inappropriate error messages produced by the REST API: when a validator client is publishing the same attestations or sync committee messages through multiple beacon nodes.
### Improvements:
* 6x speed-up in epoch processing: https://github.com/status-im/nimbus-eth2/pull/3089
* 2x speed up in Altair block processing: https://github.com/status-im/nimbus-eth2/pull/3115
* A 12% (minimum) reduction in the outgoing GossipSub traffic: https://github.com/status-im/nimbus-eth2/pull/3112
* Across the board performance improvements in the REST API: https://github.com/status-im/nimbus-eth2/pull/3092
* The REST API can now report sync committee information for the next sync period: https://github.com/status-im/nimbus-eth2/pull/3133
* Added support for the web3signer protocol (beta release):
Nimbus `v1.5.4` is a `medium-urgency` hotfix release. It addresses an important issue which, in rare cases, can lead to the loss of attestations and sync committee messages. This can, in turn, lead to a reduction in rewards.
* A rare issue during the construction of sync committee contributions: invalid BLS aggregate signatures were being produced under certain conditions; this had the potential to negatively affect the peer score of the node, and impact its ability to deliver gossip messages.
* A non-spec-compliant implementation of the `/eth/v1/validator/duties/sync/{epoch}` REST API.
* A crash in the `/eth/v2/debug/beacon/states` REST API call on systems with limited stack space.
### Improvements:
* A nice little performance improvement for block verification and replay.
* Improved error messages in the REST API.
* The `/eth/v1/config/spec` REST API now returns more information regarding spec config parameters.
## Other notable changes:
* The `--log-file` option is now deprecated and may be removed in a future release (if you wish to log to a file, we recommend redirecting the standard output).
Please note that the --log-file option was previously supported only when Nimbus was built from source. If your existing configuration used the --log-file option with a binary release, upgrading to v1.5.4 will enable the log file creation (though a deprecation warning will be printed on start-up).
Nimbus `v1.5.1` is a `high-urgency` release for all users who have already upgraded to `v1.5.0`. It fixes a deposit contract syncing issue manifesting as the warning message "Eth1 chain not synced". Under specific circumstances, such a failure to sync the Eth1 chain may result in missed Eth2 block proposals, so affected users are advised to upgrade as soon as possible.
For anyone still running Nimbus v1.4.x or earlier, migrating to v1.5.1 continues to be a `medium-urgency` but *mandatory* upgrade that must be installed **before October 27th**.
Nimbus `v1.5.0` is a `medium-urgency` but *mandatory* upgrade which adds support for the upcoming beacon chain Altair hard-fork.
Please upgrade at your earliest convenience - **before the 27th of October**.
> **N.B** You must upgrade before October 27th in order to follow the mainnet beacon chain. Failure to do so will result in downtime penalties.
This is the first hard fork for the beacon chain, and while a significant amount of testing has been conducted in the run up to this release, we recommend users regularly monitor our announcement channels ([discord]() and/or the [newsletter]()) for potential updates.
To celebrate the feature complete release candidate of our REST API, we've opened up the ports on some of our fleet nodes - don't do this at home ;)
In order to interact with these, you should point your apps and tools to:
* http://unstable.mainnet.beacon-api.nimbus.team/ - `mainnet` data, the latest `unstable` [branch](https://github.com/status-im/nimbus-eth2/tree/unstable)
* http://unstable.prater.beacon-api.nimbus.team/ - `prater` testnet data, the [latest](http://unstable.mainnet.beacon-api.nimbus.team/eth/v1/node/version) `unstable` [branch](https://github.com/status-im/nimbus-eth2/tree/unstable)
Note that right now these are very much unstable testing instances. They may be unresponsive at times - so **please do not rely on them for validation**. We may also disable them at any time.
--------------
### Notable changes
* Support for the Altair hard fork and the [latest Ethereum consensus spec](https://github.com/ethereum/consensus-specs/releases/tag/v1.1.1) (`v1.1.1`)
* Our [REST API](https://nimbus.guide/rest-api.html) is now feature complete, supporting the full [Beacon API](https://ethereum.github.io/beacon-APIs/) specification. This API should not be exposed to the public Internet as it includes multiple endpoints which could open your node to denial-of-service (DoS) attacks.
* Known limitations: To use the REST API with a validator client, you need to enable the `--subscribe-all-subnets` option. This requirement will be removed in future versions.
* A new experimental `--num-threads=X` option allows Nimbus to take advantage of multiple CPU cores when verifying attestations. Set it to `1` to use one worker thread, `2` for two worker threads, etc. The default value is set to `1` in this release, but future versions will set it to `0` (this tells the client to use as many worker threads as there are CPU cores available). N.B .**enabling anything other than `1` is considered experimental** at this stage.
* Improved peer discovery logic that ensures good connectivity to all gossip subnets.
* A new `version` metric that makes it easy to keep track of client upgrades within Grafana dashboards.
* New metrics `libp2p_peers_identity`, `libp2p_peers_traffic_read_total`, `libp2p_peers_traffic_write_total`, subdivided through Prometheus labels by the client type of the peer (i.e. nimbus, prysm, teku, lodestar or lighthouse).
* BLST has been upgraded to the latest version (`v0.3.5`).
* The `--network` option now accepts a directory with custom network metadata in the format of the [eth2-testnets](https://github.com/eth2-clients/eth2-networks) repository. The `SECONDS_PER_SLOT` configuration value remains the only setting that must be supplied at compile-time through the `-d:SECONDS_PER_SLOT=X` nim compilation flag.
This release is marked as `low-urgency` for all Nimbus users other than those who have recently updated to `v1.4.1` from a version earlier than `v1.1.0` - for these users this is a `high-urgency` release.
## Notable changes
This release fixes an issue in the upgrade procedure when upgrading from a version earlier than `1.1.0` to `1.4.x`.
**How can I tell if I've been affected?**
If you've already upgraded to `1.4.1`, you can tell that you've been affected if you're seeing the following` WRN log`:
```
Received invalid sequence of blocks
```
To re-iterate, this issue only affects users who are upgrading from `1.0.12` or earlier (released on `2021-03-10`), **and have not run any release in between**. Everyone else can ignore this release.
Nimbus `v1.4.0` users might have noticed that they are missing a small number of (seemingly random) attestations since the update. Our investigation into the matter has showed that, due to `v1.4.0`'s significant performance improvements, Nimbus validators occasionally send their first attestation for a new epoch before some peers are ready. These "slow" peers end up dropping early attestations because they're busy with the epoch transition.
It's a rare occurrence, since it requires a validator to be scheduled to attest in the first slot of an epoch *and* for the beacon node to only be connected to "slow" peers for the respective libp2p topic. If both these conditions are true, a premature attestation may be lost in time, like tears in the rain.
As a fix, we are using a larger send delay: [#2705](https://github.com/status-im/nimbus-eth2/pull/2705).
Fo those Nimbus `v1.4.0` users who are concerned about reaching optimal attestation effectiveness, we encourage you to upgrade as soon as possible.
Other changes include log flushing and metrics fixes.
Full list:
- increase attestation wait time ([#2705](https://github.com/status-im/nimbus-eth2/pull/2705))
- ensure logs are printed without delays ([#2669](https://github.com/status-im/nimbus-eth2/pull/2669))
- fix metrics on Windows ([#2707](https://github.com/status-im/nimbus-eth2/pull/2707))
This release is marked as low-urgency - please update at your convenience.
It contains **improvements to attestation effectiveness and CPU usage**.
It also contains **improvements to the RPC APIs**, as suggested by DappNode and RocketPool, and in preparation for our refactored validator client.
### Outdated Nimbus instances
A reminder that if you're running Nimbus with a version prior to `1.0.10` (March 2021) you are exposed to a vulnerability in our core cryptography library (this library is used by all eth2 clients). See this [blst security advisory](https://github.com/supranational/blst/security/advisories/GHSA-x279-68rr-jp4p) for more information. If this concerns you, please update as soon as you can.
## Notable changes
### We've added:
* Nightly builds for the very adventurous. (https://github.com/status-im/nimbus-eth2/pull/2640)
* We expect users of nightly builds to be comfortable providing debugging logs.
### We've fixed:
* RPC API endpoints (https://github.com/status-im/nimbus-eth2/pull/2585, https://github.com/status-im/nimbus-eth2/pull/2586)
*`/eth/v1/beacon/pool/attestations`
*`/api/eth/v1/validator/aggregate_and_proofs`
* Doppelganger detection: fixed false positive on fast restart (https://github.com/status-im/nimbus-eth2/pull/2656/)
### We've improved
* Database read performance improvements during epoch transitions and startup (https://github.com/status-im/nimbus-eth2/pull/2639, https://github.com/status-im/nimbus-eth2/pull/2617)
* Better usage of caches, specially when validating attestations (https://github.com/status-im/nimbus-eth2/pull/2631)
