Commit Graph

262 Commits

Author SHA1 Message Date
Etan Kissling 7c45b8f98d
convert remaining `toFork` -> `kind` for consistency (#5462)
We currently have multiple ways to obtain `ConsensusFork` or
`LcDataFork` from a forky type. Rename `toFork` to `kind`
for a consistent API naming.
2023-09-27 15:10:28 +00:00
Etan Kissling e7bc41e005
`blck` --> `forkyBlck` when using `withBlck` / `withStateAndBlck` (#5451)
For symmetry with `forkyState` when using `withState`, and to avoid
problems with shadowing of `blck` when using `withBlck` in `template`,
also rename the injected `blck` to `forkyBlck`.

- https://github.com/nim-lang/Nim/issues/22698
2023-09-21 12:49:14 +02:00
Etan Kissling 273f1d34cd
implement EIP-7514 for Deneb: Add Max Epoch Churn Limit (#5442)
* implement EIP-7514 for Deneb: Add Max Epoch Churn Limit

Cap activations per epoch according to EIP-7514:

- https://eips.ethereum.org/EIPS/eip-7514
- https://github.com/ethereum/consensus-specs/pull/3499

* apply proposer boost to first block in case of equivocation

Implement spec changes to fork choice; this only affects equivocation
when multiple blocks are signed for the same slot. Regular operation
is not changed.

- https://github.com/ethereum/consensus-specs/pull/3352

* bump test vectors to v1.4.0-beta.2-hotfix

---------

Co-authored-by: tersec <tersec@users.noreply.github.com>
2023-09-20 13:39:56 +02:00
Tanguy 4918a4e2e0
Fix direct peers (#5427)
* Fix direct peers

* Support ENRs in DP, use DP in local testnet

* fix docs

* bump libp2p
2023-09-15 18:45:55 +00:00
tersec 2b4f987c80
remove pre-v1.4.0 attestation stability subnets (#5402)
* remove pre-v1.4.0 attestation stability subnets

* re-add most of should register stability subnets on attester duties test
2023-09-11 16:03:34 +00:00
tersec cc13e0b7e0
restore full test coverage for process_rewards_and_penalties (#5407)
* restore full test coverage for process_rewards_and_penalties

* adjust ncli_db to use new iterator
2023-09-08 16:42:18 +00:00
Etan Kissling c211a3849e
remove `{.raises: [Defect].}` Nim 1.2 compatibility (#5352)
In Nim 1.6, `{.raises: [Defect].}` is no longer necessary. Remove it.
2023-08-25 11:29:07 +02:00
Etan Kissling 758c503a52
move mock genesis block hash to `tests` (#5353)
From old interop tests, a mock `eth1BlockHash` was defined in `base`.
To avoid accidental use by Nimbus, move to `tests` and rename it to
`mockEth1BlockHash`.
2023-08-25 11:28:42 +02:00
tersec d0314f0942
more ncli_db Deneb support (#5336) 2023-08-23 19:37:25 +03:00
Jacek Sieka b39e71335d
ssz: bump for 10% hash_tree_root perf increase, json loading fix (#5292)
* move file loading out of timed ncli section to focus benchmark on
deserialization rather than disk loading
* JSON loading fix included in bump, but does not practically happen
since we generally don't load hashlist from JSON
2023-08-15 13:28:33 +00:00
dependabot[bot] 6de3c49512
Bump tornado from 6.3.2 to 6.3.3 in /ncli (#5291)
Bumps [tornado](https://github.com/tornadoweb/tornado) from 6.3.2 to 6.3.3.
- [Changelog](https://github.com/tornadoweb/tornado/blob/master/docs/releases.rst)
- [Commits](https://github.com/tornadoweb/tornado/compare/v6.3.2...v6.3.3)

---
updated-dependencies:
- dependency-name: tornado
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-08-14 23:53:30 +00:00
Jacek Sieka 3c9fc49411
reduce ncli/research imports (#5275)
* reduce ncli/research imports

avoids ssl, most of chronos etc, speeds up compiles

* copyright
2023-08-10 14:52:49 +02:00
Jacek Sieka 5bc48acc36
reduce memory allocations during state transition (#5235)
This PR removes a few hundred thousand temporary seq allocations during
state transition - in particular, the flag seq was allocated per
validator while committees are computed per attestation.
2023-08-03 01:03:40 +02:00
henridf 28194468c9
Rename "data gas" to "blob gas" (#5216)
* Rename data gas to blob gas

* Update vendor/nim-eth and vendor/nim-web3
2023-08-02 22:07:57 +00:00
Jacek Sieka e8379389e7
speed up state/block loading (#5207)
* speed up state/block loading

When loading blocks and states from db/era, we currently redundantly
check their CRC32 - for a state, this costs 50ms of loading time
presently (110mb uncompressed size) on a decent laptop.

* remove `maxDecompressedDbRecordSize` - not actually used on recent
data since we store the framed format - also, we're in luck: we blew
past the limit quite some time ago
* fix obsolete exception-based error checking
* avoid `zeroMem` when reading from era store

see https://github.com/status-im/nim-snappy/pull/22 for benchmarks

* bump snappy
2023-07-26 10:47:46 +03:00
Eugene Kabanov df80ae68fa
REST server metrics and connections API endpoints. (#5193)
* Add new REST endpoints to monitor REST server connections and new chronos metrics.

* Bump head versions of chronos and presto.

* Bump chronos with regression fix.

* Remove outdated tests which was supposed to test pipeline mode.

* Disable pipeline mode in resttest.

* Update copyright year.

* Upgrade test_signing_node to start use AsyncProcess instead of std library's osproc.
Bump chronos to check graceful shutdown.

* Update AllTests.

* Bump chronos.
2023-07-21 16:54:09 +02:00
dependabot[bot] 1430981d25
Bump pygments from 2.11.2 to 2.15.0 in /ncli (#5205)
Bumps [pygments](https://github.com/pygments/pygments) from 2.11.2 to 2.15.0.
- [Release notes](https://github.com/pygments/pygments/releases)
- [Changelog](https://github.com/pygments/pygments/blob/master/CHANGES)
- [Commits](https://github.com/pygments/pygments/compare/2.11.2...2.15.0)

---
updated-dependencies:
- dependency-name: pygments
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-07-20 14:25:18 +00:00
Etan Kissling da5bb96856
avoid Nim 2.0 issue when loading config (#5202)
To avoid Nim 2.0 issue https://github.com/nim-lang/Nim/issues/22284,
explicitly specify `ref SecondarySources` instead of using `auto`,
and add `{.raises.}` annotation
2023-07-19 20:26:30 +02:00
tersec 565edfa351
remove unused code (#5158)
* remove unused code

* add copyright header
2023-07-15 16:30:52 +00:00
dependabot[bot] b3a033c66f
Bump scipy from 1.7.3 to 1.10.0 in /ncli (#5166)
Bumps [scipy](https://github.com/scipy/scipy) from 1.7.3 to 1.10.0.
- [Release notes](https://github.com/scipy/scipy/releases)
- [Commits](https://github.com/scipy/scipy/compare/v1.7.3...v1.10.0)

---
updated-dependencies:
- dependency-name: scipy
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-07-09 07:42:00 +00:00
tersec 6730e16439
replace integer cast with conversion (#5160) 2023-07-04 14:29:26 +00:00
Etan Kissling 2722778ce5
reduce `nim-eth` dependencies just for RNG (#5099)
We have several modules that import `nim-eth` for the sole purpose of
its `keys.newRng` function. This function is meanwhile a simple wrapper
around `nim-bearssl`'s `HmacDrbgContext.new()`, so the import doesn't
really serve a use anymore. Replace `keys.newRng` with the direct call
to reduce `nim-eth` imports.
2023-06-19 22:43:50 +00:00
Jacek Sieka 4ae1857898
fix missing era regeneration (#5098)
+1 is already done via a `defer`
2023-06-19 15:47:24 +00:00
tersec 939b3002e4
remove removed getDebugChainHeadsV1; fix API links in documentations (#5058) 2023-06-15 09:58:38 +00:00
tersec 1b77831758
Revert "remove removed Debug/getDebugChainHeads V1 (#4970)" (#5057)
This reverts commit 6ebcbbdf4e.
2023-06-10 08:08:01 +00:00
tersec 6ebcbbdf4e
remove removed Debug/getDebugChainHeads V1 (#4970)
* remove removed Debug/getDebugChainHeads V1

* REST API tests
2023-06-10 08:07:36 +00:00
tersec e8c6af0636
add getNextWithdrawals Beacon API (#5021) 2023-06-07 21:27:15 +00:00
tersec b25ca0833b
use v1.4.0-alpha.1 consensus spec test vectors (#5026) 2023-06-03 21:55:08 +00:00
dependabot[bot] 629e60b9ca
Bump tornado from 6.1 to 6.3.2 in /ncli (#4995)
Bumps [tornado](https://github.com/tornadoweb/tornado) from 6.1 to 6.3.2.
- [Changelog](https://github.com/tornadoweb/tornado/blob/master/docs/releases.rst)
- [Commits](https://github.com/tornadoweb/tornado/compare/v6.1.0...v6.3.2)

---
updated-dependencies:
- dependency-name: tornado
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-05-25 18:18:59 +00:00
henridf 573228ffa0
Rename eth1/ -> el/ and eth1_monitor.nim -> el_monitor.nim (#4944) 2023-05-15 05:05:12 +00:00
Etan Kissling 0701038f76
log `bls_to_execution_changes_len` in `shortLog` (#4949)
* log `bls_to_execution_changes_len` in `shortLog`

When logging blocks to console, include num BLS to Execution changes.

* fix

* lint
2023-05-13 04:29:46 +00:00
Etan Kissling 18bc47fe83
add Capella constants to REST config endpoint (#4925)
Capella constants were missing; if we want to support BLS changes from
VC, we need the new domain constant to check compatibility.
2023-05-11 12:53:02 +02:00
Etan Kissling 1ebcd8b473
`excess_data_gas` after `withdrawals` cleanup (#4900)
We already updated the field order in the actual `ExecutionPayload`,
but in init code and tests / logs etc we still used the old order.
Update those occurrences to also match the field order in the struct.
Furthermore, add `excess_data_gas` to last entry in `test_eth1_monitor`.
2023-05-05 23:15:47 +00:00
Etan Kissling 28968c11ba
remove `SAFE_SLOTS_TO_UPDATE_JUSTIFIED` (#4897)
The `SAFE_SLOTS_TO_UPDATE_JUSTIFIED` constant is no longer used as the
bouncing attack fix was removed:
https://github.com/ethereum/consensus-specs/pull/3290

Note: Some test networks still define the constant, ignoring the config
constant for now until it is no longer used.
2023-05-05 18:11:38 +00:00
Jacek Sieka 58b93ccbe0
era: Capella+ support (fixes #4752) (#4853)
Post-Capella, historical roots are computed from historical summaries
instead of being directly stored in the beacon state.

Slightly messy to pass both lists around - this is done to avoid
computing the historical root unnecessarily.
2023-04-24 15:26:28 +02:00
tersec eed34e740a
implement the getStateRandao Beacon API (#4799) 2023-04-11 18:27:48 +03:00
tersec f71a279d17
more deneb support in ncli_db and forks (#4774) 2023-03-30 10:06:23 +00:00
tersec 865d95bcbb
add capella/deneb support to ncli and clean up imports (#4771) 2023-03-29 11:34:22 +03:00
tersec e6043f656f
add Deneb support to ncli_db (#4767) 2023-03-28 16:44:38 +03:00
tersec 44b38ef137
Implement /eth/v1/validator/{beacon,sync}_committee_selections Beacon API end-points (#4760) 2023-03-22 01:47:36 +02:00
Zahary Karadjov 46f48269ef
Backwards compatible handling of the web3-url parameter in TOML 2023-03-14 17:50:03 +02:00
Etan Kissling ad118cd354
rename `stateFork` > `consensusFork` (#4718)
Just the variable, not yet `lcDataForkAtStateFork` / `atStateFork`.

- Shorten comment in `light_client.nim` to keep line width
- Do not rename `stateFork` mention in `runProposalForkchoiceUpdated`.
- Do not rename `stateFork` in `getStateField(dag.headState, fork)`

Rest is just a mechanical mass replace
2023-03-11 00:35:52 +00:00
tersec a47f0b054e
finish eip4844 to deneb module rename (#4705) 2023-03-09 01:34:17 +01:00
zah 8771e91d53
Support for driving multiple EL nodes from a single Nimbus BN (#4465)
* Support for driving multiple EL nodes from a single Nimbus BN

Full list of changes:

* Eth1Monitor has been renamed to ELManager to match its current
  responsibilities better.

* The ELManager is no longer optional in the code (it won't have
  a nil value under any circumstances).

* The support for subscribing for headers was removed as it only
  worked with WebSockets and contributed significant complexity
  while bringing only a very minor advantage.

* The `--web3-url` parameter has been deprecated in favor of a
  new `--el` parameter. The new parameter has a reasonable default
  value and supports specifying a different JWT for each connection.
  Each connection can also be configured with a different set of
  responsibilities (e.g. download deposits, validate blocks and/or
  produce blocks). On the command-line, these properties can be
  configured through URL properties stored in the #anchor part of
  the URL. In TOML files, they come with a very natural syntax
  (althrough the URL scheme is also supported).

* The previously scattered EL-related state and logic is now moved
  to `eth1_monitor.nim` (this module will be renamed to `el_manager.nim`
  in a follow-up commit). State is assigned properly either to the
  `ELManager` or the to individual `ELConnection` objects where
  appropriate.

  The ELManager executes all Engine API requests against all attached
  EL nodes, in parallel. It compares their results and if there is a
  disagreement regarding the validity of a certain payload, this is
  detected and the beacon node is protected from publishing a block
  with a potential execution layer consensus bug in it.

  The BN provides metrics per EL node for the number of successful or
  failed requests for each type Engine API requests. If an EL node
  goes offline and connectivity is resoted later, we report the
  problem and the remedy in edge-triggered fashion.

* More progress towards implementing Deneb block production in the VC
  and comparing the value of blocks produced by the EL and the builder
  API.

* Adds a Makefile target for the zhejiang testnet
2023-03-05 01:40:21 +00:00
tersec e3d96ef147
rename most eip4844Data to denebData (#4693) 2023-03-04 22:23:52 +00:00
tersec 3b41e6a0e7
rename ConsensusFork.EIP4844 to ConsensusFork.Deneb (#4692) 2023-03-04 13:35:39 +00:00
tersec 8f269c92d7
rename eip48844ImplementationMissing to denebImplementationMissing (#4654) 2023-02-23 10:37:45 +00:00
zah 6036f2e7d7
Local sim impovements (#4551)
* Local sim impovements

* Added support for running Capella and EIP-4844 simulations
  by downloading the correct version of Geth.

* Added support for using Nimbus remote signer and Web3Signer.
  Use 2 out of 3 threshold signing configuration in the mainnet
  configuration and regular remote signing in the minimal one.

* The local testnet simulation can now use a payload builder.
  This is currently not activated in CI due to lack of automated
  procedures for installing third-party relays or builders.

  You are adviced to use mergemock for now, but for most realistic
  results, we can create a simple builder based on the nimbus-eth1
  codebase that will be able to propose transactions from the regular
  network mempool.

* Start the simulation from a merged state. This would allow us
  to start removing pre-merge functionality such as the gossip
  subsciption logic. The commit also removes the merge-forcing
  hack installed after the TTD removal.

* Consolidate all the tools used in the local simulation into a
  single `ncli_testnet` binary.
2023-02-23 02:10:07 +00:00
tersec dc0bbe3a57
rm blockForkAtEpoch and switch callers to consensusForkAtEpoch (#4634) 2023-02-16 21:16:54 +01:00
Eugene Kabanov e91415662b
Keystore cache implementation. (#4372) 2023-02-16 19:25:48 +02:00