* fix REST liveness endpoint responding even when gossip is not enabled
* fix VC exit code on doppelganger hit
* fix activation epoch not being updated correctly on long deposit
queues
* fix activation epoch being set incorrectly when updating validator
* move most implementation logic to `validator_pool`, add tests
* ensure consistent logging between VC and BN
* add docs
Adds a "Slot start" log to the LC that behaves similar to BN to inform
the user that the light client is doing something, and to indicate the
latest view of the network (finalized / optimistic).
Currently, we require genesis and a checkpoint block and state to start
from an arbitrary slot - this PR relaxes this requirement so that we can
start with a state alone.
The current trusted-node-sync algorithm works by first downloading
blocks until we find an epoch aligned non-empty slot, then downloads the
state via slot.
However, current
[proposals](https://github.com/ethereum/beacon-APIs/pull/226) for
checkpointing prefer finalized state as
the main reference - this allows more simple access control and caching
on the server side - in particular, this should help checkpoint-syncing
from sources that have a fast `finalized` state download (like infura
and teku) but are slow when accessing state via slot.
Earlier versions of Nimbus will not be able to read databases created
without a checkpoint block and genesis. In most cases, backfilling makes
the database compatible except where genesis is also missing (custom
networks).
* backfill checkpoint block from libp2p instead of checkpoint source,
when doing trusted node sync
* allow starting the client without genesis / checkpoint block
* perform epoch start slot lookahead when loading tail state, so as to
deal with the case where the epoch start slot does not have a block
* replace `--blockId` with `--state-id` in TNS command line
* when replaying, also look at the parent of the last-known-block (even
if we don't have the parent block data, we can still replay from a
"parent" state) - in particular, this clears the way for implementing
state pruning
* deprecate `--finalized-checkpoint-block` option (no longer needed)
In optimistic mode, Nimbus will sync optimistically even when the
execution client is offline / not available.
An optimistic node is less secure because it has not validated block
transactions via the execution client and can thus not be used for
validation duties.
* `.era` store docs
Initial documentation for era file generation and usage
* Update docs/the_nimbus_book/src/era-store.md
Co-authored-by: zah <zahary@status.im>
* Update docs/the_nimbus_book/src/era-store.md
Co-authored-by: Kim De Mey <kim.demey@gmail.com>
* remove dupe
Co-authored-by: zah <zahary@status.im>
Co-authored-by: Kim De Mey <kim.demey@gmail.com>
The `eth1_monitor` check to require engine API from bellatrix onward
has issues in setups where the EL and CL are started simultaneously
because the EL may not be ready to answer requests by the time that the
check is performed. This can be observed, e.g., on Raspberry Pi 4 when
using Besu as the EL client. Now that the merge transition happened, the
check is also not that useful anymore, as users have other ways to know
that their setup is not working correctly (e.g., repeated exchange logs)
When the BN-embedded LC makes sync progress, pass the corresponding
execution block hash to the EL via `engine_forkchoiceUpdatedV1`.
This allows the EL to sync to wall slot while the chain DAG is behind.
Renamed `--light-client` to `--sync-light-client` for clarity, and
`--light-client-trusted-block-root` to `--trusted-block-root` for
consistency with `nimbus_light_client`.
Note that this does not work well in practice at this time:
- Geth sticks to the optimistic sync:
"Ignoring payload while snap syncing" (when passing the LC head)
"Forkchoice requested unknown head" (when updating to LC head)
- Nethermind syncs to LC head but does not report ancestors as VALID,
so the main forward sync is still stuck in optimistic mode:
"Pre-pivot block, ignored and returned Syncing"
To aid EL client teams in fixing those issues, having this available
as a hidden option is still useful.
The optimistic sync spec was updated since the LC based optsync module
was introduced. It is no longer necessary to wait for the justified
checkpoint to have execution enabled; instead, any block is okay to be
optimistically imported to the EL client, as long as its parent block
has execution enabled. Complex syncing logic has been removed, and the
LC optsync module will now follow gossip directly, reducing the latency
when using this module. Note that because this is now based on gossip
instead of using sync manager / request manager, that individual blocks
may be missed. However, EL clients should recover from this by fetching
missing blocks themselves.
Aligns the default retention policy for LC data with the one for blocks.
Minimum spec requirement for both blocks and LC data is ~5 months.
Additional use cases are better supported by retaining data for longer.
* reorganise navigation menus
* update light client guide with comparison table
* add suggested fee recipient and JWT secrets to the merge guide
* add some background info to book readme
* add JWT docs
also limit toc to make it displayable with substeps.
* Use final `v1` version for light client protocols
* Unhide LC data collection options
* Default enable LC data serving
* rm unneeded import
* Connect to EL on startup
* Add docs for LC based EL sync
`mkdocs` works with markdown similar to `mdbook` but is generally more
pleasing to the eye and has several nice UX features.
This PR does the bulk of the transition - likely, a followup would be
needed to fully make use of the extra features and navigation.
Book pages have been kept url-compatible, meaning that for the most
part, old links should continue to work!
Co-authored-by: Etan Kissling <etan@status.im>
* update docs
* introduce mdbook-admonish for nice looking callouts
* new section on data directory
* recommend source build for advanced users and direct the rest to
binaries
* more strongly highlight that execution client is needed
* write an actual deposit guide
* remove cruft / fix links / etc
* remove web3 url prompt in launcher script
The interactive prompt for web3 has outlived its utility as we now load
url:s from command line params and config files, preventing the prompt
from correctly detecting when it's needed.
Also, after the merge, a JWT secret will (likely) be needed.
* log notice when web3 url is missing
* fix docs to not mention default that doesn't exist
* fix scripts to properly quote arguments
* systemd: use same service file as package
* systemd: recommend setting up a user
* clean up obsolete / ephemeral notes
* eth1 -> execution, in many places
* clean up docker instructions to match the experience of running the
runner script
* add basic instructions / links for execution clients other than geth
* clear a few more references to JSON-RPC, update to REST
* update hardware requirements with execution client extras
* start talking about merge requirements (this needs a dedicated page)
* note that binaries do exist, where relevant
* list gitcoin under donations
* update syncing docs
* `NOT` is for `NOTICE` in logs
* update rpi hardware recommendations
* prevent automatic doppelganger restarts in systemd service file
Validator monitoring improves logging by giving more specific monitoring
information, and can now be seen as complete.
Previously, logging has focused on "Attestation sent" messages which
carry little informational value when things go wrong, and are overly
aggressive when everything works as expected (sending attestations is
the norm).
* lower "Attestation sent" log to `INFO`
* mark 1.7.0 as the start of the validator monitor feature - previous
versions had significant bugs in totals mode
The `pyrmont` testnet has been discontinued.
For experiments, it's still possible to run pyrmont nodes by passing a
genesis/config, but this PR removes the bundled `--network:pyrmont`
option.
* update docs
* remove empty docs
* remove obsolete `eth2-stats` page
When performing trusted node sync, historical access is limited to
states after the checkpoint.
Reindexing restores full historical access by replaying historical
blocks against the state and storing snapshots in the database.
The process can be initiated or resumed at any point in time.
* book: add trusted node sync to index
...and some doc updates
* Update docs/the_nimbus_book/src/start-syncing.md
Co-authored-by: Ștefan Talpalaru <stefantalpalaru@yahoo.com>
Co-authored-by: Ștefan Talpalaru <stefantalpalaru@yahoo.com>
I found this documentation a little confusing because it says "you must" leading me to think there may be some kind of dire consequences for NOT having it synced before starting the validator (in the context of importing existing keys from another setup, not starting a new validator with a new deposit). Modifying the language to make the consequences of not having the syncing done clear.
* Trusted node sync
Trusted node sync, aka checkpoint sync, allows syncing tyhe chain from a
trusted node instead of relying on a full sync from genesis.
Features include:
* sync from any slot, including the latest finalized slot
* backfill blocks either from the REST api (default) or p2p (#3263)
Future improvements:
* top up blocks between head in database and some other node - this
makes for an efficient backup tool
* recreate historical state to enable historical queries
* fixes
* load genesis from network metadata
* check checkpoint block root against state
* fix invalid block root in rest json decoding
* odds and ends
* retry looking for epoch-boundary checkpoint blocks
* min viable changes
* make clear keymanager api is not ready for mainnet
* Update docs/the_nimbus_book/src/keymanager-api.md
Co-authored-by: Jacek Sieka <jacek@status.im>
* Update docs/the_nimbus_book/src/keymanager-api.md
Co-authored-by: zah <zahary@gmail.com>
Co-authored-by: Jacek Sieka <jacek@status.im>
Co-authored-by: zah <zahary@gmail.com>
* REST cleanups
* reject out-of-range committee requests
* print all hex values as lower-case
* allow requesting state information by head state root
* turn `DomainType` into array (follow spec)
* `uint_to_bytesXX` -> `uint_to_bytes` (follow spec)
* fix wrong dependent root in `/eth/v1/validator/duties/proposer/`
* update documentation - `--subscribe-all-subnets` is no longer needed
when using the REST interface with validator clients
* more fixes
* common helpers for dependent block
* remove test rules obsoleted by more strict epoch tests
* fix trailing commas
* Update docs/the_nimbus_book/src/rest-api.md
* Update docs/the_nimbus_book/src/rest-api.md
Co-authored-by: sacha <sacha@status.im>
* Fix REST some rest call signatures and implement a simple API benchmark tool
* Implement #3129 (Optimized history traversals in the REST API)
Other notable changes:
The `updateStateData` procedure in the `blockchain_dag.nim` module is
optimized to not rewind down to the last snapshot state saved in the
database if the supplied input state can be used as a starting point
instead.
* Disallow await in withStateForBlockSlot
Validator monitoring based on and mostly compatible with the
implementation in Lighthouse - tracks additional logs and metrics for
specified validators so as to stay on top on performance.
The implementation works more or less the following way:
* Validator pubkeys are singled out for monitoring - these can be
running on the node or not
* For every action that the validator takes, we record steps in the
process such as messages being seen on the network or published in the
API
* When the dust settles at the end of an epoch, we report the
information from one epoch before that, which coincides with the
balances being updated - this is a tradeoff between being correct
(waiting for finalization) and providing relevant information in a
timely manner)
* Support starting from altair
* hide `finalized-checkpoint-` - they are incomplete and usage may cause
crashes
* remove genesis detection code (broken, obsolete)
* enable starting ChainDAG from altair checkpoints - this is a
prerequisite for checkpoint sync (TODO: backfill)
* tighten checkpoint state conditions
* show error when starting from checkpoint with existing database (not
supported)
* print rest-compatible JSON in ncli/state_sim
* altair/merge support in ncli
* more altair/merge support in ncli_db
* pre-load header to speed up loading
* fix forked block decoding
* Reduce the logging-related breaking changes
* Don't disable the stdout log when the `--log-file` option is used
* Rename `--log-stdout` to `--log-format` and hide it
* Deprecate the `--log-file` option
zah
Jacek Sieka
cheny0
tersec
Chen Yang
Etan Kissling
Kim De Mey
Julian Ste
Shiti Saxena
Miguel Tenorio
omahs
Miran
zah
0kok0
sacha
Ștefan Talpalaru
TennisBowling
Paweł Bylica
Jeremy Schlatter
yslcrypto
Zachinquarantine
lbly
rasberrylasagna