Commit Graph

444 Commits

Author SHA1 Message Date
zah 8771e91d53
Support for driving multiple EL nodes from a single Nimbus BN (#4465)
* Support for driving multiple EL nodes from a single Nimbus BN

Full list of changes:

* Eth1Monitor has been renamed to ELManager to match its current
  responsibilities better.

* The ELManager is no longer optional in the code (it won't have
  a nil value under any circumstances).

* The support for subscribing for headers was removed as it only
  worked with WebSockets and contributed significant complexity
  while bringing only a very minor advantage.

* The `--web3-url` parameter has been deprecated in favor of a
  new `--el` parameter. The new parameter has a reasonable default
  value and supports specifying a different JWT for each connection.
  Each connection can also be configured with a different set of
  responsibilities (e.g. download deposits, validate blocks and/or
  produce blocks). On the command-line, these properties can be
  configured through URL properties stored in the #anchor part of
  the URL. In TOML files, they come with a very natural syntax
  (althrough the URL scheme is also supported).

* The previously scattered EL-related state and logic is now moved
  to `eth1_monitor.nim` (this module will be renamed to `el_manager.nim`
  in a follow-up commit). State is assigned properly either to the
  `ELManager` or the to individual `ELConnection` objects where
  appropriate.

  The ELManager executes all Engine API requests against all attached
  EL nodes, in parallel. It compares their results and if there is a
  disagreement regarding the validity of a certain payload, this is
  detected and the beacon node is protected from publishing a block
  with a potential execution layer consensus bug in it.

  The BN provides metrics per EL node for the number of successful or
  failed requests for each type Engine API requests. If an EL node
  goes offline and connectivity is resoted later, we report the
  problem and the remedy in edge-triggered fashion.

* More progress towards implementing Deneb block production in the VC
  and comparing the value of blocks produced by the EL and the builder
  API.

* Adds a Makefile target for the zhejiang testnet
2023-03-05 01:40:21 +00:00
Jacek Sieka daf8ee6c59
docs: add apt repo (#4644)
* docs: add apt repo

* Update docs/the_nimbus_book/src/keep-updated.md

Co-authored-by: tersec <tersec@users.noreply.github.com>

* more package manager integration

* fix admonition

---------

Co-authored-by: zah <zahary@status.im>
Co-authored-by: tersec <tersec@users.noreply.github.com>
2023-03-02 17:22:07 +01:00
tersec 29fb65a9db
automated update of v1.3.0-rc.2 to v1.3.0-rc.3 consensus spec URLs (#4647) 2023-02-21 16:43:21 +00:00
Zahary Karadjov 8202d4a84e
Version 23.2.0 2023-02-16 20:19:22 +02:00
zah ff464e49cf
Implement the set of gas_limit end-points in the Keymanager API (#4612)
Fixes #3946
2023-02-15 15:10:31 +00:00
Jacek Sieka 822c339607
e2store: avoid uint48 (#4625)
In SSZ, `uint32` is used for offsets, effectively limiting the size of
an SSZ entry to 2**32 bytes.

Also, `uint48` isn't a valid SSZ type, so the header was not correctly
defined according to the SSZ spec - the extra 2 bytes are left for
future expansion instead.
2023-02-15 14:51:17 +01:00
tersec 4c845b9749
note 8GB requirement is for building; fix macOS supported version (#4594) 2023-02-06 08:21:11 +01:00
Andy Tudhope caa9eec394
Update hardware.md (#4584)
Due to increased requirements when building beacon nodes.

Though this is annoying, I still think you all are wonderful and magical beings.
2023-02-02 18:14:54 +01:00
tersec 58ed9308d2
automated v1.3.0-rc.1 to v1.3.0-rc.2 consensus spec URL updates (#4568) 2023-01-31 00:26:57 +01:00
tersec 8c2e01da24
consensus spec ref URL updates to v1.3.0-rc.1 (#4534) 2023-01-20 22:33:05 +00:00
tersec 073c544f0c
automated update from v1.3.0-rc.0 to v1.3.0-rc.1 consensus spec URLs (#4517) 2023-01-17 16:10:52 +00:00
Jacek Sieka d8caab500d
Enable validator monitor by default (#4468)
By enabling the validator monitor, more precise information about the
lifecycle of an attestation is logged at the higher `NOTICE` log level
while current `sent` messages are logged at `INF` instead, since they
are less interesting.

In particular, missed attestations and those that vote for the wrong
head are now detected and logged at NOTICE.

In addition to logging, this feature enables rich metrics around
attestation and sync committee performance - by default, validators are
tracked in aggregate but a detailed mode exists as well

This feature has been available since early Nimbus days, but it has now
been tuned and optimised such that it is safe to enable by default, even
for large setups.

* enable automatic validator monitoring by default
* replace `--validator-monitor-totals` flag with
`--validator-monitor-details` - the detailed mode is disabled by default
* lower "sent" log level to `INF` for several messages - in particular
those that are traced by the validator monitor

This is a retake on #3531 which was later reverted in #3578.
2023-01-16 11:28:35 +01:00
Etan Kissling 2324136552
add `LightClientHeader` wrapper (#4481)
The light client data structures were changed to accommodate additional
fields in future forks (e.g., to also hold execution data).

There is a minor change to the JSON serialization, where the `header`
properties are now nested inside a `LightClientHeader`.
The SSZ serialization remains compatible.

See https://github.com/ethereum/consensus-specs/pull/3190
and https://github.com/ethereum/beacon-APIs/pull/287
2023-01-13 16:46:35 +01:00
tersec e28e1aeec8
a few consensus spec ref URL updates (#4483) 2023-01-10 16:14:17 +00:00
tersec 2dd3cd786f
consensus spec ref URL update v1.3.0-{alpha.2,rc.0}; copyright year update (#4477) 2023-01-09 22:44:44 +00:00
Jacek Sieka 0ba9fc4ede
History pruning (fixes #4419) (#4445)
Introduce (optional) pruning of historical data - a pruned node will
continue to answer queries for historical data up to
`MIN_EPOCHS_FOR_BLOCK_REQUESTS` epochs, or roughly 5 months, capping
typical database usage at around 60-70gb.

To enable pruning, add `--history=prune` to the command line - on the
first start, old data will be cleared (which may take a while) - after
that, data is pruned continuously.

When pruning an existing database, the database will not shrink -
instead, the freed space is recycled as the node continues to run - to
free up space, perform a trusted node sync with a fresh database.

When switching on archive mode in a pruned node, history is retained
from that point onwards.

History pruning is scheduled to be enabled by default in a future
release.

In this PR, `minimal` mode from #4419 is not implemented meaning
retention periods for states and blocks are always the same - depending
on user demand, a future PR may implement `minimal` as well.
2023-01-07 10:02:15 +00:00
tersec c5d1683f19
spec ref URL & copyright year updates (#4467) 2023-01-06 16:28:46 +00:00
Jacek Sieka 7c2ed5c609
Always-on optimistic mode (#4458)
With https://github.com/status-im/nimbus-eth2/pull/4420 implemented, the
checks that we perform are equivalent to those of a `SYNCING` EL - as
such, we can treat missing EL the same as SYNCING and proceed with an
optimistic sync.

This mode of operation significantly speeds up recovery after an offline
EL event because the CL is already synced and can immediately inform the
EL of the latest head.

It also allows using a beacon node for consensus archival queries
without an execution client.

* deprecate `--optimistic` flag
* log block details on EL error, soften log level because we can now
continue to operate
* `UnviableFork` -> `Invalid` when block hash verification fails -
failed hash verification is not a fork-related block issue
2023-01-04 15:51:14 +00:00
tersec 9e699fd2fc
manual rebase of Bump wheel from 0.37.1 to 0.38.1 in /docs #4446 (#4456) 2023-01-03 18:02:05 +00:00
zah 0be57eec6e
Don't package the nimbus_validator_client in the nimbus-eth2 docker images; Document the new dedicated image (#4441) 2022-12-20 23:32:31 +02:00
zah 94a87da4a2
Documentation for the --with-deposit-snapshot option (#4440) 2022-12-20 23:32:13 +02:00
tersec bb4ea37baa
update EF consensus spec URLs from v1.3.0-alpha.1 to v1.3.0-alpha.2 (#4432) 2022-12-15 12:15:12 +00:00
tersec ad8e682f76
remove redundant justification and finalization tests (#4412) 2022-12-09 22:45:48 +00:00
Jacek Sieka 6e2a02466e
unify bn/vc doppelganger detection (#4398)
* fix REST liveness endpoint responding even when gossip is not enabled
* fix VC exit code on doppelganger hit
* fix activation epoch not being updated correctly on long deposit
queues
* fix activation epoch being set incorrectly when updating validator
* move most implementation logic to `validator_pool`, add tests
* ensure consistent logging between VC and BN
* add docs
2022-12-09 17:05:55 +01:00
cheny0 6b5682df70
change all :s to s (#4403)
* change all :s to s

* edit one more :s
2022-12-09 10:42:52 +01:00
tersec 415b11aa67
EIP4844 tweaks to pass SSZ consensus object tests (#4390) 2022-12-05 21:36:53 +00:00
tersec 7e5f40e04c
remove built-in Ropsten support (#4280)
* remove built-in Ropsten support

* link to ropsten shutdown-specific blog post

Co-authored-by: zah <zahary@status.im>
2022-12-05 11:15:00 +02:00
Zahary Karadjov 7dd4718735
Version 22.11.0 2022-12-01 07:38:54 +02:00
Chen 50d53e77c7 Minor edits for practicing Github workflow 2022-11-25 10:49:42 +08:00
tersec 61c5ac32d8
automated consensus spec ref URL update to v1.3.0-alpha.1 (#4354) 2022-11-24 19:07:02 +00:00
Chen Yang 925843c35b 4347 fix the REST API navigation typo 2022-11-24 18:00:48 +08:00
Jacek Sieka 44cab3a4cc
Update quick-start.md 2022-11-23 15:41:39 +01:00
Etan Kissling 2823be7306
periodically log LC sync progress (#4339)
Adds a "Slot start" log to the LC that behaves similar to BN to inform
the user that the light client is doing something, and to indicate the
latest view of the network (finalized / optimistic).
2022-11-22 16:39:03 +01:00
Kim De Mey 84bacf7cc6
Nimbus guide: Fix link to consensus light client page (#4336) 2022-11-18 18:22:10 +01:00
Jacek Sieka 1b90f65f7b
doc updates (#4314)
* optimise index / landing page real estate, add feature list
* fee recipient in quickstart
* recommend http over websockets  (fewer command line options needed)
* simplify PBS, fee recipient docs
* add VC role / sentry node docs (#4140)
* Update docs/the_nimbus_book/src/rest-api.md

Co-authored-by: tersec <tersec@users.noreply.github.com>
2022-11-13 07:59:55 +01:00
tersec 909c095e64
initial automated v1.2.0 -> v1.3.0-alpha.0 consensus spec URL update (#4296) 2022-11-08 02:37:28 +00:00
Jacek Sieka d839b9d07e
State-only checkpoint state startup (#4251)
Currently, we require genesis and a checkpoint block and state to start
from an arbitrary slot - this PR relaxes this requirement so that we can
start with a state alone.

The current trusted-node-sync algorithm works by first downloading
blocks until we find an epoch aligned non-empty slot, then downloads the
state via slot.

However, current
[proposals](https://github.com/ethereum/beacon-APIs/pull/226) for
checkpointing prefer finalized state as
the main reference - this allows more simple access control and caching
on the server side - in particular, this should help checkpoint-syncing
from sources that have a fast `finalized` state download (like infura
and teku) but are slow when accessing state via slot.

Earlier versions of Nimbus will not be able to read databases created
without a checkpoint block and genesis. In most cases, backfilling makes
the database compatible except where genesis is also missing (custom
networks).

* backfill checkpoint block from libp2p instead of checkpoint source,
when doing trusted node sync
* allow starting the client without genesis / checkpoint block
* perform epoch start slot lookahead when loading tail state, so as to
deal with the case where the epoch start slot does not have a block
* replace `--blockId` with `--state-id` in TNS command line
* when replaying, also look at the parent of the last-known-block (even
if we don't have the parent block data, we can still replay from a
"parent" state) - in particular, this clears the way for implementing
state pruning
* deprecate `--finalized-checkpoint-block` option (no longer needed)
2022-11-02 10:02:38 +00:00
Jacek Sieka a958313feb
remove obsolete troubleshooting section 2022-10-27 14:15:09 +02:00
Jacek Sieka b08d0ff2ab
Optimistic mode (#4262)
In optimistic mode, Nimbus will sync optimistically even when the
execution client is offline / not available.

An optimistic node is less secure because it has not validated block
transactions via the execution client and can thus not be used for
validation duties.
2022-10-26 20:44:45 +00:00
Jacek Sieka e6a346d471
era: document block root access 2022-10-26 18:34:39 +02:00
tersec 9045840c75
point to Goerli/Prater launchpad requiring much less ETH (#4257) 2022-10-26 11:03:37 +03:00
Jacek Sieka 5e3e4b1706
whitespace 2022-10-25 11:14:07 +02:00
Jacek Sieka d00a3f1dae
Use SSZ object to define header 2022-10-25 11:13:33 +02:00
Jacek Sieka 22d68de365
`.era` store docs (#4234)
* `.era` store docs

Initial documentation for era file generation and usage

* Update docs/the_nimbus_book/src/era-store.md

Co-authored-by: zah <zahary@status.im>

* Update docs/the_nimbus_book/src/era-store.md

Co-authored-by: Kim De Mey <kim.demey@gmail.com>

* remove dupe

Co-authored-by: zah <zahary@status.im>
Co-authored-by: Kim De Mey <kim.demey@gmail.com>
2022-10-14 14:28:37 +02:00
tersec b8e9240ee0
the merge has occurred (#4219)
* the merge has occurred

* restore 🐼
2022-10-05 17:04:13 +03:00
tersec ce915c0a03
add beaconcha.in mainnet relay overview to guide (#4217) 2022-10-04 19:45:06 +03:00
tersec 76b3db3d7e
add prater.beaconcha.in external block builder overview link (#4211)
* add prater.beaconcha.in external block builder overview link

* Update docs/the_nimbus_book/src/external-block-builder.md

Co-authored-by: Etan Kissling <etan@status.im>

Co-authored-by: Etan Kissling <etan@status.im>
2022-10-03 16:56:49 +00:00
Julian Ste 86800066e4
Typo and updated specific version (#4203)
* Typo and updated specific version

* Update keep-updated.md

* Update keep-updated.md
2022-10-01 20:20:11 +00:00
Julian Ste bee50d7578
Updated nomenclature for eth2 naming (#4202) 2022-10-01 20:01:41 +00:00
tersec 6c18e82506
link to document of available external builder relays (#4157) 2022-09-26 19:11:34 +00:00