Commit Graph

439 Commits

Author SHA1 Message Date
tersec d1941b670a
refactor payload builder REST client usage (#4973)
* refactor payload builder REST client usage

* change HTTP response code
2023-05-25 18:38:56 +03:00
Etan Kissling 00728e9bb7
use `executionValid` bit in `BlockRef` (#4956)
Update `beacon_node/rpc` usage for the new `executionValid` field.
2023-05-25 15:57:24 +02:00
tersec cd087b9a43
replace `optimisticRoots` table with field in `BlockRef` (#4969)
* replace optimisticRoots table with field in BlockRef

* copyright year

* mark finalized blocks as verified on load

* Update beacon_chain/consensus_object_pools/block_dag.nim

Co-authored-by: Etan Kissling <etan@status.im>

* expand non-optimistic block checking to all pre-merge blocks; refactor markBlockVerified to use BlockRef rather than block root and remove superfluous caller in newPayload path replaced by addResolvedHeadBlock BlockRef construction

* don't treat finalized block specially; VALID status is sticky

---------

Co-authored-by: Etan Kissling <etan@status.im>
2023-05-20 12:18:51 +00:00
henridf 1cf777c64b
Fix sync for blocks older than MIN_EPOCHS_FOR_BLOB_SIDECARS_REQUESTS (#4977)
When doing sync for blocks older than
MIN_EPOCHS_FOR_BLOB_SIDECARS_REQUESTS, we skip the blobs by range
request, but we then pass en empty blob sequence to
validation, which then fails.

To fix this: Use an Option[Blobsidecars] to allow expressing the
distinction between "empty blob sequence" and "blobs unavailable". Use
the latter for "old" blocks, and don't attempt to run blob validation.
2023-05-19 16:25:11 +00:00
Zahary Karadjov 4c3850f7df
Disable the use of incbin as it breaks the retail build on Linux/ARM 2023-05-19 04:08:02 +03:00
Etan Kissling 40e89937c5
segregate sync committee messages by period / fork (#4953)
`SyncCommitteeMsgPool` grouped messages by their `beacon_block_root`.
This is problematic around sync committee period boundaries and forks.
Around sync committee period boundaries, members from both the current
and next sync committee may sign the same `beacon_block_root`; mixing
the signatures from both committees together is a mistake. Likewise,
around fork transitions, the `signing_root` changes, so those messages
also need to be segregated.
2023-05-17 07:55:55 +03:00
Etan Kissling adcabf9ad7
cleanup gossip message handler setup (#4961)
* cleanup gossip message handler setup

Reduce duplication and make gossip handler setup easier to read.

* explicit returns for async validators
2023-05-16 09:46:41 +02:00
Jacek Sieka 51418a7894
Incremental pruning (#4887)
* Incremental pruning

When turning on pruning the first time the current pruning algorithm
will prune the full database at startup. This delays restart
unnecessarily, since all of the pruned space is not needed at once.

This PR introduces incremental pruning such that we will never prune
more than 32 blocks or the sync speed, whichever is higher.

This mode is expected to become default in a follow-up release.
2023-05-12 13:37:15 +03:00
Jacek Sieka b3c6320d56
embed genesis states using incbin (#4905) 2023-05-11 11:11:00 +00:00
henridf 5dfd814588
Load trusted setup (#4870)
* Kzg: Load trusted setup

* scripts/launch_local_testnet.sh: set FIELD_ELEMENTS_PER_BLOB

* Use right setup file for mainnet/minimal

* Force rebuild

* Add comment explaining why build with -f
2023-05-11 11:52:44 +03:00
henridf be3f5b1eac
More blob tweaks/fixes from running in devnet (#4933)
* BeaconNode: don't call fetchMissingblobs with empty list

* More logging

* BlockProcessor.checkBloblessSignature: Add missing return value
2023-05-11 00:36:35 +00:00
henridf 23adf15e5a
Blob handling sync fixes (#4888)
* Fix groupBlobs

* Fix getShortMap

* Fix blob handling in sync

* lint

* Add some blob-related logging
2023-05-06 08:58:50 +00:00
Etan Kissling 508819633a
more consensus-specs v1.3.0 bumps (#4898)
Updates more consensus-specs references to v1.3.0.
This PR only includes updates for sections otherwise unmodified.
2023-05-05 20:38:51 +00:00
Etan Kissling e408c7573d
bump more consensus-specs refs to v1.3.0 (with comments) (#4901)
Bump some more consensus-specs links to v1.3.0, and also align comments
with latest specs where they differ, and fix typos.
2023-05-05 20:37:56 +00:00
henridf ef0b95dfbc
RequestManager: add support for fetching Blobs (#4844)
* RequestManager: add support for fetching Blobs

* Review feedback

* Lint

* Change peekSortedBlobless -> peekBlobless
2023-04-28 12:57:35 +00:00
henridf 12d640b691
Request manager: Handle blobless blocks (#4833)
Post-Deneb, when the request manager receives a missing block from a
peer, it needs to check if the corresponding blobs are available, and
if so pass them along. If they aren't available, the newly-fetched
block must be put in blobless quarantine (while the blobs are
retrieved, coming in next commit).
2023-04-19 19:37:38 +03:00
Etan Kissling e6e4ba9de6
clean up redundant tests and config (#4836)
The consensus-spec-tests already cover the scenarios of our custom test
runner, so the custom tests can be removed. Also cleans up unused config
flags and related unreachable logic.
2023-04-18 21:26:36 +02:00
henridf 8f2f40f0af
Remove blockBlobsVerifier (#4829)
Having a separate 'blockBlobsVerifier' function for post-Deneb blocks
is no longer of any benefit. This commit removes it.
2023-04-18 02:12:57 +02:00
Etan Kissling cb9e0eed49
allow trusted node sync based on LC trusted block root (#4736)
* allow trusted node sync based on LC trusted block root

Extends `trustedNodeSync` with a new `--trusted-block-root` option that
allows initializing a light client. No `--state-id` must be provided.
The beacon node will then use this light client to obtain the latest
finalized state from the remote server in a trust-minimized fashion.
Note that the provided `--trusted-block-root` should be somewhat recent,
and that security precautions such as comparing the state root against
block explorers is still recommended.

* fix

* workaround for `valueOr` limitations

* reduce magic numbers

* digest len > context len for readability

* move `cstring` conversion to caller

* avoid abbreviations

* `return` codestyle
2023-04-16 06:07:07 +00:00
tersec ae45e11f8e
log when attempting to validate with exited validators (#4812) 2023-04-15 21:31:24 +00:00
henridf 021de18e06
Quarantine and reassembly of gossiped blobs and blocks (#4808) 2023-04-13 19:11:40 +00:00
Etan Kissling 566c178e59
add ANSI art for Deneb 🐟 (#4803)
Trigger ANSI art on upgrade to Deneb

Art created by http://beatscribe.com (beatscribe#1008 on Discord)
2023-04-10 12:42:19 +00:00
tersec 742de1347d
only add active validators to knownValidators at startup (#4793) 2023-04-06 13:53:06 +00:00
henridf 04302081b4
Add blob sidecar gossip validation (#4785)
* Add blob gossip validation

* lint

* Add test for getBlobSidecarTopic

* Fix closure variable capture issue

* Update beacon_chain/nimbus_beacon_node.nim

Co-authored-by: tersec <tersec@users.noreply.github.com>

---------

Co-authored-by: tersec <tersec@users.noreply.github.com>
2023-04-04 15:12:34 +00:00
Zahary Karadjov 4d1b2dd9f5
Merge branch 'stable' into unstable 2023-03-17 17:51:39 +02:00
Zahary Karadjov ee610cbf34
Restore the hidden --web3-force-polling option as a deprecated one 2023-03-14 18:54:15 +02:00
henridf 9906a0d5a2
Gate Deneb `beacon_block` subscription (#4735) 2023-03-14 11:52:01 +00:00
Etan Kissling b0cccae7f4
rename `(with|at)StateFork` > `(with|at)ConsensusFork` (#4725) 2023-03-11 14:39:29 +00:00
Etan Kissling ad118cd354
rename `stateFork` > `consensusFork` (#4718)
Just the variable, not yet `lcDataForkAtStateFork` / `atStateFork`.

- Shorten comment in `light_client.nim` to keep line width
- Do not rename `stateFork` mention in `runProposalForkchoiceUpdated`.
- Do not rename `stateFork` in `getStateField(dag.headState, fork)`

Rest is just a mechanical mass replace
2023-03-11 00:35:52 +00:00
Etan Kissling 969c6f73ae
misc local `EIP4844` > `Deneb` bumps (#4717)
* misc local `EIP4844` > `Deneb` bumps

* fix
2023-03-11 00:28:19 +00:00
Etan Kissling e4b5478fff
rename `ForkDigests.eip4844` > `ForkDigests.deneb` (#4715) 2023-03-10 17:13:40 +00:00
tersec a47f0b054e
finish eip4844 to deneb module rename (#4705) 2023-03-09 01:34:17 +01:00
henridf 90640cce05
Update sync to use post-decoupling RPC (#4701)
* Update sync to use post-decoupling RPCs

blob_sidecars_by_range returns a flat list of sidecars, which must
then be grouped per-slot.

* Add test for groupBlobs

* createBlobs: convert proc to func
2023-03-07 20:19:17 +00:00
zah 8771e91d53
Support for driving multiple EL nodes from a single Nimbus BN (#4465)
* Support for driving multiple EL nodes from a single Nimbus BN

Full list of changes:

* Eth1Monitor has been renamed to ELManager to match its current
  responsibilities better.

* The ELManager is no longer optional in the code (it won't have
  a nil value under any circumstances).

* The support for subscribing for headers was removed as it only
  worked with WebSockets and contributed significant complexity
  while bringing only a very minor advantage.

* The `--web3-url` parameter has been deprecated in favor of a
  new `--el` parameter. The new parameter has a reasonable default
  value and supports specifying a different JWT for each connection.
  Each connection can also be configured with a different set of
  responsibilities (e.g. download deposits, validate blocks and/or
  produce blocks). On the command-line, these properties can be
  configured through URL properties stored in the #anchor part of
  the URL. In TOML files, they come with a very natural syntax
  (althrough the URL scheme is also supported).

* The previously scattered EL-related state and logic is now moved
  to `eth1_monitor.nim` (this module will be renamed to `el_manager.nim`
  in a follow-up commit). State is assigned properly either to the
  `ELManager` or the to individual `ELConnection` objects where
  appropriate.

  The ELManager executes all Engine API requests against all attached
  EL nodes, in parallel. It compares their results and if there is a
  disagreement regarding the validity of a certain payload, this is
  detected and the beacon node is protected from publishing a block
  with a potential execution layer consensus bug in it.

  The BN provides metrics per EL node for the number of successful or
  failed requests for each type Engine API requests. If an EL node
  goes offline and connectivity is resoted later, we report the
  problem and the remedy in edge-triggered fashion.

* More progress towards implementing Deneb block production in the VC
  and comparing the value of blocks produced by the EL and the builder
  API.

* Adds a Makefile target for the zhejiang testnet
2023-03-05 01:40:21 +00:00
tersec d058aa09c8
more withdrowls (#4674) 2023-03-02 17:13:35 +01:00
henridf 5a9e63a029
Remove uses of beacon_block_and_blobs_sidecar topic (#4682)
* Remove use of beacon_block_and_blobs_sidecar topic

This topic goes away with decoupled blocks and blobs.

* remove use of getBeaconBlockAndBlobsSidecarTopic from test

* update nimbus_light_client.nim
2023-03-01 21:30:20 +01:00
henridf 3681177cf4
Remove ForkySignedBeaconBlockMaybeBlobs (#4681)
This commit removes ForkySignedBeaconBlockMaybeBlobs and all
references. I tried to pull that thread only as little as was needed
to get rid of it. Left a placeholder BlobSidecar array (in lieu of
Opt[BlobsSidecar]) in a few places; this will be used as we rebuild
the decoupled implementation.
2023-02-28 11:36:17 +00:00
henridf a999da3361
Remove beacon_block_and_blobs_sidecar_by_root RPC (#4672)
Along with its use in the request manager.
2023-02-26 23:17:40 +00:00
tersec c6cefd8914
add success logging for being configured to use payload builder (#4653) 2023-02-23 11:46:18 +02:00
zah 6036f2e7d7
Local sim impovements (#4551)
* Local sim impovements

* Added support for running Capella and EIP-4844 simulations
  by downloading the correct version of Geth.

* Added support for using Nimbus remote signer and Web3Signer.
  Use 2 out of 3 threshold signing configuration in the mainnet
  configuration and regular remote signing in the minimal one.

* The local testnet simulation can now use a payload builder.
  This is currently not activated in CI due to lack of automated
  procedures for installing third-party relays or builders.

  You are adviced to use mergemock for now, but for most realistic
  results, we can create a simple builder based on the nimbus-eth1
  codebase that will be able to propose transactions from the regular
  network mempool.

* Start the simulation from a merged state. This would allow us
  to start removing pre-merge functionality such as the gossip
  subsciption logic. The commit also removes the merge-forcing
  hack installed after the TTD removal.

* Consolidate all the tools used in the local simulation into a
  single `ncli_testnet` binary.
2023-02-23 02:10:07 +00:00
tersec 29fb65a9db
automated update of v1.3.0-rc.2 to v1.3.0-rc.3 consensus spec URLs (#4647) 2023-02-21 16:43:21 +00:00
Jacek Sieka 83f9745df1
restore doppelganger check on connectivity loss (#4616)
* restore doppelganger check on connectivity loss

https://github.com/status-im/nimbus-eth2/pull/4398 introduced a
regression in functionality where doppelganger detection would not be
rerun during connectivity loss. This PR reintroduces this check and
makes some adjustments to the implementation to simplify the code flow
for both BN and VC.

* track when check was last performed for each validator (to deal with
late-added validators)
* track when we performed a doppel-detectable activity (attesting) so as
to avoid false positives
* remove nodeStart special case (this should be treated the same as
adding a validator dynamically just after startup)

* allow sync committee duties in doppelganger period

* don't trigger doppelganger when registering duties

* fix crash when expected index response is missing

* fix missing slashingSafe propagation
2023-02-20 13:28:56 +02:00
tersec 0cc0c7e6b7
builder spec v0.3.0 URL updates (#4641)
* builder spec v0.3.0 URL updates

* remove stray non-URL/comment change
2023-02-18 01:54:30 +01:00
tersec a382498cfe
batch-verify BLS to execution change messages (#4637) 2023-02-17 13:35:12 +00:00
Eugene Kabanov e91415662b
Keystore cache implementation. (#4372) 2023-02-16 19:25:48 +02:00
tersec e342fdd97a
rename stateForkAtEpoch to consensusForkAtEpoch (#4627) 2023-02-16 11:32:12 +02:00
zah ff464e49cf
Implement the set of gas_limit end-points in the Keymanager API (#4612)
Fixes #3946
2023-02-15 15:10:31 +00:00
zah 067ba13c52
Capella metadata for Sepolia (#4615)
Other changes:

Renamed the `EIP_4844_FORK_*` config constants to `DENEB_FORK_*` as
this matches the latest spec and it's already used in the official
Sepolia config.
2023-02-15 14:44:09 +00:00
Etan Kissling 1ac7f1a47a
rm unused `lowParticipation` flag (#4624)
Remove an unused flag and associated (unreachable) logic, as the flag
wasn't even passed to that module.
2023-02-15 00:25:22 +01:00
henridf 59e41dc65d
EIP4844 sync (#4581)
* EIP4844 Sync

* Pass eip4844 fork epoch rather than cfg to syncmanager

* Fix sync

* Update test

* map->mapIt
2023-02-11 20:48:35 +00:00