Commit Graph

4229 Commits

Author SHA1 Message Date
Etan Kissling 0cca65904d
reduce `ProveField` warnings in `conf.nim` (#5485)
Address some verbose `ProveField` warnings and `XCannotRaiseY` hints
in `conf.nim`.
2023-10-05 17:15:03 +03:00
Etan Kissling dd1f362536
align `Beacon(Block(Body)?|State)Type` with other fork sugar (#5483)
The templates for `BeaconBlock`, `BeaconBlockBody` and `BeaconState`
are the only ones using a `macro` mechanism for code generation.
This prevents using the dot-syntax style `consensusFork.BeaconFoo`
in some situations, and also tends to trigger naming conflicts,
requiring the `Type` suffix. Furthermore, the `macro` only works
for types that are re-defined in every single `ConsensusFork`.

Replacing with the simpler but more verbose approach used for other
types for consistency and to avoid the downsides of the `macro`.

Furthermore, simplify `test_fixture_sanity_blocks` to use `forks` sugar.
2023-10-05 14:01:40 +02:00
tersec 89686fa274
avoid needlessly rescoring aggregates during block attestation packing (#5471) 2023-10-04 22:38:10 +00:00
Etan Kissling 297c768816
reduce redundant zero initialization for LC data objects (#5479)
Directly initialize `ForkedLightClientObj` instead of separately first
 setting the `kind` (initializing everything to zero) and then assigning
the forky data after that.
2023-10-04 18:11:45 +02:00
Etan Kissling 37967ba03b
`ProveField` cleanups in `slashing_protection_common` (#5477)
Cleanup of `ProveField` warnings in `slashing_protection_common` module.
Note that `ProveField` is disabled by default in makefile, but sometimes
these pop up when doing a regular `nim c`, and cleaning these may allow
enabling the warning in some future.

As the `case` is over `a.kind`, `ProveField` warnings for `b` have to be
suppressed using `{.push.}` / `{.pop.}` for comparison operators.
2023-10-04 10:26:52 +02:00
Etan Kissling 24c68bbcf7
fix `ProveField` warning in `network_metadata` (#5476)
Spurious `ProveField` warning can be avoided by using `case` instead of
`if` on `metadata.genesis.kind`. Also suppress `GlobalVar` hints when
`incbinEnabled` is used, which has global `let` definitions.
2023-10-04 10:19:05 +02:00
Etan Kissling bfe76be4d7
debug log duration for processing attestation queue (#5473)
Depending on the load, processing of queued attestations may take a
while. Log processing duration at debug level.
2023-10-03 22:17:42 +02:00
Etan Kissling e1a8e9a2fa
rm unused `GenesisStateRef` (#5469)
In `el_manager`, there was an unused `GenesisStateRef` type. Remove it.
2023-10-02 19:00:23 +00:00
Jacek Sieka 9a48e2e922
fix stale activity log (#5465) 2023-09-30 13:31:08 +00:00
Etan Kissling 8aaea1a14e
simplify fork choice test (#5467)
Reduce overly verbose fork branching logic in fork choice test with
generic implementation based on `consensusFork`.
2023-09-29 13:26:34 +00:00
Jacek Sieka 252473d2d1
ssz: bump (#5463)
more `zeroMem` avoidance, value semantics for merkleizer
2023-09-28 11:59:33 +00:00
Zahary Karadjov 77d6bc5f42
Merge stable into unstable 2023-09-28 11:32:54 +01:00
tersec 7ecc1096e3
document strange-looking capella/bellatrix apparent mismatch in block building (#5461) 2023-09-27 16:06:22 +00:00
Etan Kissling 7c45b8f98d
convert remaining `toFork` -> `kind` for consistency (#5462)
We currently have multiple ways to obtain `ConsensusFork` or
`LcDataFork` from a forky type. Rename `toFork` to `kind`
for a consistent API naming.
2023-09-27 15:10:28 +00:00
Eugene Kabanov 4fb95d000d
REST server fixes and improvements. (#5422)
* Move from Option[T] to Opt[T] usage.

* Add `finalized` flag.

* Fix compilation issue.

* Http415 error code for some REST API calls.
Introduce more comprehensive error reporting for block calls.
Deprecate decodeEthConsensusVersion() function.

* Bump http-utils.

* Fix copyright year.

* Fix serialization issue.

* Address review comments.

* Post rebase fixes.
2023-09-27 16:45:33 +02:00
Zahary Karadjov cfa0268d89
Version 23.9.1
Changes:

* Updated network metadata for Holesky
* Use hash_tree_root instead of SHA256 when verifying the Holesky
  genesis state
2023-09-25 21:24:25 +01:00
tersec 710f26745d
small attestation pool clean up (#5458) 2023-09-24 08:50:48 +00:00
Eugene Kabanov 98e08a9e02
Fix cancellation deprecate warnings. (#5454)
Add noCancel in exception handlers.
Refactor pending cancellations code.
2023-09-24 09:28:09 +02:00
Eugene Kabanov e3fe762ec9
Fix unknown fields should be ignored when parsing liveness responses. (#5418) 2023-09-23 08:44:54 +02:00
Etan Kissling 47639ef89c
replace `Eth2NetworkMetadata` case object with `Result` (#5452)
`Eth2NetworkMetadata` has an `incompatible` case to hold an error string
in case the loaded file is not compatible with the compile-time config.
The same can be modeled with a `Result[Eth2NetworkMetadata, string]` and
avoids followup checks for the `incompatible` case.
2023-09-22 23:45:24 +02:00
Etan Kissling 5150505e66
bump `nim-ssz-serialization` to `a5767c1b4992dca3e5bd72557aadd4df0f183bf8` (#5457)
- nimble/ci updates
- use common super-type in `raisesssz` pragma
2023-09-22 23:44:57 +02:00
Eugene Kabanov caaad4777c
BN+LC+SN: Fix cancellation deprecate warnings. (#5455)
* Fix deprecation cancel() warnings in SN, BN, LC.

* Fix missing import.
2023-09-22 11:06:27 +00:00
Eugene Kabanov 88565e6da4
Make KeystoreCache working with KeystoreManagement API. (#5443) 2023-09-21 20:07:02 +02:00
tersec 2895a9a05c
automated consensus spec URL updating to v1.4.0-beta.2 (#5453) 2023-09-21 18:06:51 +00:00
Jacek Sieka e64b31986e
Increase default file limit on startup (#5436)
We use file descriptors for validators and sockets and might run out of
either on high-validator setups - increasing the limit here is harmless
and avoids a common limiting factor in setup

Co-authored-by: Etan Kissling <etan@status.im>
2023-09-21 16:08:13 +02:00
Etan Kissling e7bc41e005
`blck` --> `forkyBlck` when using `withBlck` / `withStateAndBlck` (#5451)
For symmetry with `forkyState` when using `withState`, and to avoid
problems with shadowing of `blck` when using `withBlck` in `template`,
also rename the injected `blck` to `forkyBlck`.

- https://github.com/nim-lang/Nim/issues/22698
2023-09-21 12:49:14 +02:00
tersec 351472a2d4
per payload-builder payload builder validator registration (#5431) 2023-09-20 17:00:37 +00:00
Etan Kissling ee57a07411
implement Confirmation rule prerequisite - fork choice filter change (#5450)
To support confirmation rule via beacon-APIs as described in spec PR,
add `--debug-fork-choice-version=pr3431` option and enable when Deneb
fork is scheduled. To opt-out, `--debug-fork-choice-version=stable`,
or don't schedule Deneb.

- https://github.com/ethereum/consensus-specs/pull/3431
- https://github.com/ethereum/consensus-specs/issues/3466

"will bundle this with deneb release":

- https://github.com/ethereum/pm/issues/844#issuecomment-1673359012
2023-09-20 18:27:55 +02:00
Etan Kissling 273f1d34cd
implement EIP-7514 for Deneb: Add Max Epoch Churn Limit (#5442)
* implement EIP-7514 for Deneb: Add Max Epoch Churn Limit

Cap activations per epoch according to EIP-7514:

- https://eips.ethereum.org/EIPS/eip-7514
- https://github.com/ethereum/consensus-specs/pull/3499

* apply proposer boost to first block in case of equivocation

Implement spec changes to fork choice; this only affects equivocation
when multiple blocks are signed for the same slot. Regular operation
is not changed.

- https://github.com/ethereum/consensus-specs/pull/3352

* bump test vectors to v1.4.0-beta.2-hotfix

---------

Co-authored-by: tersec <tersec@users.noreply.github.com>
2023-09-20 13:39:56 +02:00
Etan Kissling 1640c45a95
reject invalid proofs when validating blobs (#5445)
Currently, passing `0xc00000...` proof seems to pass `verifyProofs`.
Unsure why such a check is not necessary in spec, and also unsure
whether it is correct to reject proof at infinity, or if it could
occur, e.g., for a blob containing all 0 bytes. Weird overall...

* proper fix
2023-09-19 22:13:58 +00:00
Etan Kissling c1b43d166b
avoid repeated total active balance computation in fork choice (#5437)
We currently compute `justified_total_active_balance` inside
`calculateProposerBoost`, despite that sum already being known
in the `EpochRef` cache. Tracking `justified_total_active_balance`
whenever the justified checkpoint updates allows replacing the
repeated computation with a lookup, at minimal memory cost.
2023-09-19 22:04:55 +02:00
tersec ee75d45a8b
fix `ATTESTATION_PROPAGATION_SLOT_RANGE` unknown constant warning (#5433)
* fix ATTESTATION_PROPAGATION_SLOT_RANGE unknown constant warning

* verify underflow not an issue
2023-09-19 06:54:58 +00:00
Tanguy 4918a4e2e0
Fix direct peers (#5427)
* Fix direct peers

* Support ENRs in DP, use DP in local testnet

* fix docs

* bump libp2p
2023-09-15 18:45:55 +00:00
Eugene Kabanov 12ef43bcc0
Fix issue when VC could not be interrupted while its in pre-genesis period. (#5424) 2023-09-15 12:39:26 +02:00
tersec 325d5a6afa
log builder unblinding requests (#5434) 2023-09-15 09:27:38 +00:00
tersec dae4beb4a2
log engine getPayload requests (#5432) 2023-09-15 08:56:32 +00:00
Eugene Kabanov aec953e4da
VC: Fix VC must not crash if beacon node address could not be resolved. (#5388)
* Fix VC should not crash, if beacon node URL could not be resolved.

* Bump chronos.

* Update .gitmodules.
2023-09-15 07:34:32 +02:00
tersec 5a29ad7e4f
update some consensus-spec URLs to v1.4.0-beta.1 (#5429) 2023-09-14 18:23:59 +00:00
tersec 7a001d4896
remove 4 more missing constants in preset handling warning (#5428) 2023-09-14 12:37:35 +00:00
Etan Kissling 176ea09c2b
cleanup `OnBlockAdded` usage (#5426)
Reduce repetitiveness when using forked `OnBlockAdded` callbacks by
introducing a template to obtain appropriate cb from `ConsensusFork`.
2023-09-13 17:57:54 +00:00
Etan Kissling d7aaf1b809
only request `nextPeriod` sync duties close to end of `currentPeriod` (#5409)
Currently we always request duties for current and next sync period.
As sync periods are quite long (~27 hrs on Mainnet), having access to
the duties so early doesn't help too much. To avoid running into errors
when the BN does not have the duties available around period boundary,
delay requesting them until the current period is close to finish.

`SYNC_COMMITTEE_SUBNET_COUNT` epochs are what the spec says should be
the lookahead timing of starting to subscribe to sync committee gossip.
Reusing the constant here for consistency.

This fixes these warning messages in the first slot of a new period.

```
rocketpool_validator   | WRN 2023-09-07 20:19:35.439+00:00 Beacon node has incompatible configuration reason="Epoch value is far from the future;400;getSyncCommitteeDuties(first);invalid-request" node=http://eth2:5052[Nimbus/v23.8.0-872b19-stateofus] node_index=0 node_roles=AGBSDT
rocketpool_validator   | WRN 2023-09-07 20:19:35.440+00:00 Unable to get sync committee duties        period=889 epoch=227584 reason="Epoch value is far from the future;400;getSyncCommitteeDuties(first);invalid-request" service=duties_service
rocketpool_validator   | NOT 2023-09-07 20:19:35.441+00:00 Beacon node is in sync                     head_slot=7274495 sync_distance=1 is_optimistic=false node=http://eth2:5052[Nimbus/v23.8.0-872b19-stateofus] node_index=0 node_roles=AGBSDT
```
2023-09-13 08:58:28 +02:00
Etan Kissling 4700030231
fix sync contribution validation for first slot per period (#5408)
Sync committee duties are performed by the sync committee as determined
by slot + 1. We did it correctly for individual messages, but selected
the incorrect participants for aggregate contributions for the very
first slot per period (roughly 1 per ~27 hrs on Mainnet). The faulty
participants selection code was originally introduced in #2925.
2023-09-13 03:23:18 +02:00
andri lim 5c88e74c08
Bump stint to v2.0: new array backend (#5113)
* bump stint to v2.0: new array backend

* Fix missing isEven in libnimbus_lc.nim

* bump nim-stint: compiles with arm64
2023-09-13 01:16:04 +07:00
Etan Kissling 8fa5580c50
add infrastructure to select fork choice version (#5387)
To allow testing https://github.com/ethereum/consensus-specs/issues/3466
add support for selecting fork choice version at launch. This means we
can deploy a different logic when `DENEB_FORK_EPOCH != FAR_FUTURE_EPOCH`
that won't be used on Mainnet.
2023-09-12 09:52:51 +02:00
Zahary Karadjov 6b1ab81046
Support both snappy-encoded and pure SSZ genesis states 2023-09-12 01:52:15 +03:00
tersec 2b4f987c80
remove pre-v1.4.0 attestation stability subnets (#5402)
* remove pre-v1.4.0 attestation stability subnets

* re-add most of should register stability subnets on attester duties test
2023-09-11 16:03:34 +00:00
Etan Kissling ff176a1569
fix `syncCommitteeDuties` log in `Slot end` message (#5416)
In `Slot end`, sync duties are reported one slot too late,
because sync committee duties are determined by the next slot
instead of the current one. Address that by taking account of this.
2023-09-11 10:51:55 +00:00
Eugene Kabanov 3c3c4e8edf
Fix /eth/v1/validator/liveness/{epoch} call names and links in code (#5392)
* Fix getLiveness() call links and names.

* Add getLiveness() decoding test.

* Remove BN state updates from getLiveness() call.

* Fix AllTests.

* Reduce number of tests.
2023-09-11 12:07:34 +02:00
tersec 10ec7be686
optimize epoch registry processing (#5412) 2023-09-11 11:21:50 +02:00
tersec cc13e0b7e0
restore full test coverage for process_rewards_and_penalties (#5407)
* restore full test coverage for process_rewards_and_penalties

* adjust ncli_db to use new iterator
2023-09-08 16:42:18 +00:00
Zahary Karadjov f0bb1cbd4e
Version 23.9.0 2023-09-08 13:33:51 +03:00
Zahary Karadjov c7db7d0f2c
Add CLI options `--genesis-state` and `--genesis-state-url` 2023-09-08 13:10:18 +03:00
zah 53589b5a7d
Add metadata for the Holesky network (#5337)
* Add metadata for the Holesky network

* Add copyright banner to the new Nim module

* Working version

* Bump Chronos to fix downloading from Github
* Add checksum check of the downloaded file
* Clean up debugging code and obsolete imports
2023-09-08 08:53:27 +03:00
tersec f53422c181
optimize epoch transition via get_flag_index_deltas() and get_inactivity_penalty_deltas() (#5404)
* optimize epoch transition via get_flag_index_deltas() and get_inactivity_penalty_deltas()

* directly mutate Altair info in {get,update}_flag_and_inactivity_deltas
2023-09-07 11:14:52 +00:00
tersec fbcb476ff9
allow archiving blobs (#5391) 2023-09-07 08:19:33 +02:00
zah 2b5bd74e15
Rename --validator-source to --web3-signer-url and document it (#5389)
Also allows multiple instances to be configured
2023-09-06 22:04:10 +03:00
Etan Kissling 89d133d0e5
further reduce stack size of LC helpers for Nim 2.0 (#5401)
The `is_next_sync_committee_known` helper allocates a fresh
`SyncCommittee` in the caller and `nimZeroMem`s it on each use.
Use `static` to compare against a compile-time zeroed copy instead.
This also should help reduce stack size far enough to link with Nim 2.0.
2023-09-06 16:05:12 +02:00
Etan Kissling ac3b2b4233
rename `U` to `maxLen` in `statediff` for Nim 2.0 (#5396)
Nim 2.0 gets confused when compiling `all_tests`:

```
Error: undeclared identifier: 'maxLen'
candidates (edit distance, scope distance); see '--spellSuggest':
 (3, 7): 'Table'
 (3, 7): 'len'
 (3, 7): 'max'
```

Renaming the generic parameter `U` to `maxLen` fixes this somehow.
It also increases readability to use the same name consistently.
2023-09-06 02:39:21 +00:00
Eugene Kabanov 4ce8e77b56
Address issue #5362. (#5370)
* Address issue #5362.

* Remove RestSpec object declaration and getSpec() API call declaration.

* Address review comments.
2023-09-05 23:39:01 +03:00
Etan Kissling 8ffb80e954
avoid `Taskpool.spawn` within `{.async.}` for Nim 2.0 (#5393)
In Nim 2.0, attempting to use `Taskpool.spawn` inside `{.async.}` `proc`
leads to `Error: cannot generate destructor for generic type: Isolated`.

Add an intermediate wrapper `proc` that performs the `spawn` operation
to workaround the problem.
2023-09-05 21:36:35 +02:00
Etan Kissling dbb1a63ca9
collapse `partialBeaconBlock` templates into one (#5386)
Have a single `Forky` template for `partialBeaconBlock` production
rather than have several copies that are mostly identical.
2023-09-05 15:59:17 +02:00
tersec 0987628c0b
simplify beacon validator sync committee production for blocks slightly (#5364) 2023-09-05 12:31:33 +02:00
Eugene Kabanov afb5fdae34
Dynamic validators loop (#5385)
* Add periodic dynamic validators polling.
* Add dynamic validators tests.
2023-09-04 22:14:58 +03:00
Etan Kissling be9ecfa1c9
add helpers for processing withdrawals to `libnimbus_lc.a` (#5374)
Similar to the existing helpers for processing transactions / receipts,
extend `libnimbus_lc.a` with support for processing withdrawals as well.
2023-09-04 20:44:03 +02:00
tersec 41ce581149
invalidate only required htr cache portion for immutable validators (#5381) 2023-09-04 19:48:31 +02:00
Jacek Sieka bd513ca2ba
separate CL/EL timing in add block log (#5380)
the metric remains the same in order not to change overall meaning, but
in the logs we can do better - often, the EL is at the core of a
slowdown.
2023-09-04 20:23:29 +03:00
tersec a8c56b1660
update some consensus-spec URLs to v1.4.0-beta.1 (#5379) 2023-09-01 09:31:52 +00:00
Eugene Kabanov 757328372a
Dynamic validators set. (#5366)
* Initial commit.

* Fix argument to be optional.

* Adopt options.md.
2023-08-31 15:16:15 +03:00
tersec 8e413d9abb
rename --stop-at-epoch flag to --debug-stop-at-epoch (#5361) 2023-08-30 14:15:19 +03:00
tersec 6c42cfa79e
use correct decimal or hex encoding/decoding for REST GetSpec (#5371) 2023-08-30 12:43:25 +03:00
tersec f2d3859d80
ensure blob quarantine insertion always succeeds (#5369) 2023-08-29 16:46:25 +00:00
Etan Kissling 09020ebd2f
add helpers for processing receipts to `libnimbus_lc.a` (#5360)
Similar to the existing helpers for processing transactions,
extend `libnimbus_lc.a` with support for processing receipts as well.
2023-08-28 15:56:40 +00:00
Jacek Sieka 778a209d09
move ctrl-c handler out of exported function (#5347)
otherwise it gets set also for unit tests that callit which interferes
with gdb.
2023-08-28 10:40:23 +02:00
tersec 29dbab916c
don't prematurely process blocks waiting for blobs; fix cosmetic head block opt/non-opt logging (#5363) 2023-08-27 07:45:24 +00:00
tersec db6f4e8090
update some consensus-spec URLs to v1.4.0-beta.1 (#5357) 2023-08-25 15:58:44 +00:00
Etan Kissling 8ff0f623b4
change unused template to unused func (#5355)
The intentionally unused `toSszType` template may trigger warnings
as it is unused. Replacing it with an `{.error.}` `func` prevents that.
2023-08-25 11:09:13 +00:00
Etan Kissling d8f44ed905
fix LC header validation in Deneb (#5350)
`blob_gas_used` must be 0 before Deneb.
2023-08-25 10:36:10 +00:00
Etan Kissling 991c31f42b
add helpers for processing transactions to `libnimbus_lc.a` (#5269)
It is useful to verify transactions data against `transactionsRoot`.
Add corresponding functionality to the light client library.
2023-08-25 11:29:39 +02:00
Etan Kissling c211a3849e
remove `{.raises: [Defect].}` Nim 1.2 compatibility (#5352)
In Nim 1.6, `{.raises: [Defect].}` is no longer necessary. Remove it.
2023-08-25 11:29:07 +02:00
Etan Kissling 9637d3f8d0
remove unused `type` in `makeLimitedUInt` (#5354)
The `type L` definition inside `makeLimitedUInt` is unused. Remove it.
2023-08-25 11:28:52 +02:00
Etan Kissling 758c503a52
move mock genesis block hash to `tests` (#5353)
From old interop tests, a mock `eth1BlockHash` was defined in `base`.
To avoid accidental use by Nimbus, move to `tests` and rename it to
`mockEth1BlockHash`.
2023-08-25 11:28:42 +02:00
Tanguy 36413c8f21
Bump libp2p for gossipsub improvements (#5229)
* Bump libp2p

* Add bandwidthEstimate param

* Fix typo and type

* Add config for light client

* Fix unit

* Add debug prefix

---------

Co-authored-by: Jacek Sieka <jacek@status.im>
Co-authored-by: diegomrsantos <diegomrsantos@gmail.com>
Co-authored-by: Diego <diego@status.im>
2023-08-24 16:04:19 +02:00
Zahary Karadjov d262109011
Version 23.8.0 2023-08-24 14:19:21 +03:00
tersec b11443291b
refactor BN engine/builder block selection to work as REST block v3 backend (#5345) 2023-08-24 02:02:30 +00:00
tersec 152dd74179
propagate newPayload-VALID to block ancestors (#5343) 2023-08-23 19:56:35 +00:00
tersec ef184c8959
subscribe to Deneb blob subnets (#5342) 2023-08-23 19:31:41 +00:00
Jacek Sieka a5151bc546
`validator_duties`->`beacon_validators` (#5333)
This PR renames the existing `validator_duties` to `beacon_validators`
and in doing so, names validators running inside the beacon node process
"beacon validators" while those running the VC can be referred to as
"client validators" to disambiguate the two.

The existing `validator_duties` instead takes on a new responsibility:
as a home for logic shared between beacon and client validators - ie
code that provides consistency in implementation and behavior between
the two modes of operation.

Not only does this simplify reasoning about where to put code -it also
reduces the number of dependencies the validator client has from ~5000
to ~3000 modules (!) according to `nim genDepend` significantly reducing
compile times.
2023-08-23 19:39:57 +03:00
tersec ec30b0cc77
rm Bellatrix builder API support remnant (#5340) 2023-08-23 19:35:48 +03:00
tersec c08d125fe3
treat more constants as quasi-runtime-presets per v1.4.0 consensus specs (#5335) 2023-08-21 19:58:51 +00:00
tersec 52640e9e03
fix/remove remaining fcuV2 in Deneb (#5332) 2023-08-20 17:35:37 +00:00
tersec d889b62b79
remove assumption of shared finalized-block ancestery between fork choice and chaindag in getDebugForkChoice (#5328) 2023-08-20 07:19:07 +00:00
Etan Kissling 2dae8ea26e
cleanup `{.raises.}` for `writeValue` (#5308)
- Remove unnecessary `Defect` references
- Remove spurious `SerializationError` references
- Remove duplicate `writeValue` template in `keystore.nim`;
  same implementation already exists a bit further above in same file.
2023-08-19 17:11:56 +02:00
Etan Kissling 8de71bee83
allow filtering block processor logs by topic (#5321)
Add separate log topic for `block_processor` messages.

Topic named similar to the other `_processor` modules:

- `eth2_processor` --> `gossip_eth2`
- `light_client_processor` --> `gossip_lc`
- `optimistic_processor` --> `gossip_opt`
2023-08-19 11:17:29 +00:00
Etan Kissling 8e500aa62f
allow filtering fee recipient logs by topic (#5320)
Add separate log topic for `Refreshing fee recipient` messages.
2023-08-19 11:16:30 +00:00
tersec 6fd4983f6b
EIP-4788/parentBeaconBlockRoot fixes (#5318) 2023-08-19 08:38:17 +00:00
tersec 2d2d5bd549
fix blob sent/not sent logging (#5316) 2023-08-18 19:21:40 +00:00
Etan Kissling ff037f1375
remove unnecessary `mixin` (#5310)
Nim 2.0 gets confused with the `mixin self, id` in `validator_monitor`.
It looks like this is actually not needed, so removing it.
2023-08-18 15:55:06 +02:00
Etan Kissling 9a4a5d0f88
ensure LC runs `exchangeTransitionConfiguration` loop (#5312)
In #4465, a regression was introduced by deleting the periodic call to
`engine_exchangeTransitionConfiguration` in the Nimbus light client.
This led to the light client never treating the EL as online and,
subsequently, not sending `engine_newPayload` requests for new blocks.
Strangely, `engine_forkchoiceUpdated` requests still make it through :)

Geth still requires both `engine_newPayload` and `fcU` to be called.
By restoring the `exchangeTransitionConfiguration` loop, `newPayload`
requests are once more issued to the EL.
2023-08-18 11:30:02 +02:00
tersec 38b1bafd87
switch Nim issue reference to still-open non-dup of issue (#5314) 2023-08-18 08:13:31 +00:00
Etan Kissling 8064f5bcce
explicitly mention which `Success` enum to use (#5309)
In Nim 2.0 it can no longer infer that we mean `ResponseCode.Success`
inside `eth2_network`. Explicitly refer to that type to work around.
2023-08-17 18:33:35 +00:00
Etan Kissling 072ec1f0d2
explicitly indicate type of `nil` in LC db `disposeSafe` (#5305)
In Nim 2.0, compiler fails to infer the type of `nil` within the
`disposeSafe` template. Avoid that problem by explicitly indicating it.
2023-08-17 17:46:58 +00:00
Etan Kissling 13f436bc24
explicit `int` array indices for Nim 2.0 (#5306)
In Nim, when using a `distinct` capacity for an array, `int` lookups
don't work. Therefore, define the array with `int` capacity.
2023-08-17 17:26:08 +00:00
Jacek Sieka 49729e1ef3
prevent concurrent `storeBlock` calls (fixes #5285) (#5295)
When a block is introduced to the system both via REST and gossip at the
same time, we will call `storeBlock` from two locations leading to a
dupliace check race condition as we wait for the EL.

This issue may manifest in particular when using an external block
builder that itself publishes the block onto the gossip network.

* refactor enqueue flow
* simplify calling `addBlock`
* complete request manager verifier future for blobless blocks
* re-verify parent conditions before adding block

among other things, it might have gone stale or finalized between one
call and the other
2023-08-17 15:12:37 +02:00
Jacek Sieka 5d936c24e4
fix batch validation when `--num-threads=1` (#5303)
without a dedicated taskpools thread, validation will never happen!
2023-08-17 10:58:13 +02:00
tersec 0d1614326d
16-byte align ValidatorSig and TrustedSig (#5301) 2023-08-17 06:27:26 +00:00
tersec 22b7b721d0
adjust digest hashing not to care about alignment; increase ValidatorPubKey alignment (#5296) 2023-08-17 03:06:21 +00:00
Eugene Kabanov 4a84457376
VC: Fix API timeouts issue. (#5297)
* Fix VC timeouts issue.
Add tests.

* Update AllTests.

* Remove assertion, which could crash VC when API call was interrupted by some higher logic.
2023-08-16 16:14:39 +00:00
tersec 230626306a
use fcU V3 in Deneb/Cancun (#5293) 2023-08-15 23:00:35 +00:00
tersec c3584a9ea7
fix regression in remote signing REST schema (#5294) 2023-08-15 22:39:12 +00:00
Eugene Kabanov 025b0e3f04
VC: Refactor sync committee duties and selection proofs calculation. (#5184)
* Initial commit.

* Fix issues in fill*() procedures.

* Enable sync committee processing.

* Address review comments.

* Address remaining review comments.

* Add sync committee selection proof pruning mechanism.

* Fix recursion issue.
2023-08-15 19:00:27 +02:00
Eugene Kabanov 255131d331
Bump chronos. (#5281)
Add current/last connection query to the connection information.
2023-08-15 13:14:34 +02:00
tersec af37a96dbd
don't send fcUs every block if in lc-opt regime and block putatively finalized (#5248) 2023-08-15 09:27:56 +00:00
Jacek Sieka f77548310f
avoid quadratic attestation queue iteration (#5288)
every attestation is processed with a new wall time so we end up
iterating over all attestations for every attestation we queue - this is
4% of cpu time on a subscribe-all-subnets node

* remove redundant zero checks - block root must be an existing block
and therefore cannot be zero
* simplify "hasn't-voted" check to root only (isZeroMemory is dubiously
implemented for objects)
2023-08-14 16:48:30 +03:00
tersec d171303133
update some consensus spec URLs to v1.4.0-beta.1 (#5287) 2023-08-12 10:38:06 +00:00
tersec 6cd8f3ba55
Update constants/presets for consistency with consensus-specs v1.4.0 (#5284) 2023-08-12 03:10:12 +00:00
henridf 9efd26c2e9
Add support for POST /eth/v2/beacon/blocks (#5214)
* Add support for POST /eth/v2/beacon/blocks

* More descriptive errors

* Address review feedback

* Return 500 (not 400) for a missing implementation case
2023-08-12 03:08:54 +00:00
henridf 9ceed40090
Prune blobs (#5283)
* Prune blobs

* Address review feedback
2023-08-11 00:15:34 +00:00
Jacek Sieka 3c9fc49411
reduce ncli/research imports (#5275)
* reduce ncli/research imports

avoids ssl, most of chronos etc, speeds up compiles

* copyright
2023-08-10 14:52:49 +02:00
Etan Kissling 973e9e9a98
speed up LC signature verification (#5268)
We know the aggregate publickey of a fully participating sync committee.
Because participation is typically very high (>95%), it is faster to
start from that aggregate publickey and subtract the individual keys of
non-participants, than summing up all the participating pubkeys.
2023-08-09 08:50:07 +02:00
tersec 85e1976ac3
automated consensus spec URL updating to v1.4.0-beta.1 (#5280) 2023-08-09 03:58:47 +00:00
tersec 8c3d1cd3aa
ensure `ValidatorPubKey` int-compatibly aligned to fix UB (#5277)
* ensure ValidatorPubKey int-compatibly aligned

* use sizeof(Hash) explicitly
2023-08-08 19:01:32 +00:00
tersec f8520c8b1c
use v1.4.0 stability subnets by default (#5276) 2023-08-08 19:01:04 +00:00
tersec 18061f7991
Remove most remaining portions of Bellatrix Builder API (#5272) 2023-08-08 00:05:17 +00:00
Etan Kissling 6a2bac5cee
need even more log lines for debugging keymanager (#5260)
Keymanager test logs are still cut off, further increase log lines.
2023-08-07 18:02:08 +02:00
tersec b9920a3a59
don't overwrite immutable parts of already-present validators (#5264) 2023-08-07 14:52:42 +00:00
Etan Kissling d7afa1c78a
add helpers for processing EL block header to `libnimbus_lc.a` (#5199)
To obtain the correct `transactions_root` and `withdrawals_root`,
it is necessary to process execution block header. Light client updates
don't contain the correct MPT roots.
2023-08-07 14:23:44 +02:00
tersec 2c424cac87
Merkle is someone's name (#5265) 2023-08-07 10:06:47 +00:00
tersec d97027fb43
remove less-than-useful warning (#5245)
* remove less-than-useful warning

* remove redundant check for node.config.payloadBuilderEnable
2023-08-06 15:12:32 +00:00
tersec 2ae2d0a042
remove Bellatrix Builder API (Signed)BuilderBid (#5262)
* remove Bellatrix Builder API (Signed)BuilderBid

* stub out BlindedBeaconBlock
2023-08-06 09:20:43 +00:00
Etan Kissling 5b70a686e3
further adjust `test_keymanager_api` logs (#5259)
It is still unclear how `test_keymanager_api` sometimes fails in CI;
further adjust logging parameters.
2023-08-05 13:12:44 +00:00
Etan Kissling ee00fb3d80
replace copy with symlink (#5252)
One of the test files is accidentally a copy instead of a symlink.
2023-08-05 11:06:55 +00:00
Etan Kissling e9940115e4
switch from `create`/`dealloc` to `new+GC_ref`/`GC_unref` (#5238)
Using `create` with objects containing managed objects is broken:
https://github.com/nim-lang/Nim/issues/22341

Switch to a safer pattern based on `new+GC_ref`/`GC_unref` instead.
2023-08-05 09:26:27 +02:00
tersec 1f5dd83fbd
unplumbing Bellatrix builder API support (#5203) 2023-08-05 05:13:08 +00:00
tersec d542410a17
remove more Bellatrix-only Builder API support infrastructure (#5251) 2023-08-05 00:11:51 +00:00
tersec 001fbd0970
Revert "Revert "correctly align genesis size (#5244)" (#5253)" (#5254)
This reverts commit 26dcee111b.
2023-08-04 20:49:12 +00:00
tersec 26dcee111b
Revert "correctly align genesis size (#5244)" (#5253)
This reverts commit 706ce01bb1.
2023-08-04 20:11:20 +00:00
Jacek Sieka 706ce01bb1
correctly align genesis size (#5244) 2023-08-04 18:08:15 +03:00
tersec 1c55ef7345
note summary data about invalid baked-in states (#5247) 2023-08-04 07:36:42 +02:00
Etan Kissling 50dfd846f0
fix inconsistent capitalization (#5240)
Two cases of inconsistent capitalization, fix them.
2023-08-03 18:53:32 +02:00
Jacek Sieka b8a32419b8
async batch verification (+40% sig verification throughput) (#5176)
* async batch verification

When batch verification is done, the main thread is blocked reducing
concurrency.

With this PR, the new thread signalling primitive in chronos is used to
offload the full batch verification process to a separate thread
allowing the main threads to continue async operations while the other
threads verify signatures.

Similar to previous behavior, the number of ongoing batch verifications
is capped to prevent runaway resource usage.

In addition to the asynchronous processing, 3 addition changes help
drive throughput:

* A loop is used for batch accumulation: this prevents a stampede of
small batches in eager mode where both the eager and the scheduled batch
runner would pick batches off the queue, prematurely picking "fresh"
batches off the queue
* An additional small wait is introduced for small batches - this helps
create slightly larger batches which make better used of the increased
concurrency
* Up to 2 batches are scheduled to the threadpool during high pressure,
reducing startup latency for the threads

Together, these changes increase attestation verification throughput
under load up to 30%.

* fixup

* Update submodules

* fix blst build issues (and a PIC warning)

* bump

---------

Co-authored-by: Zahary Karadjov <zahary@gmail.com>
2023-08-03 11:36:45 +03:00
Jacek Sieka 5bc48acc36
reduce memory allocations during state transition (#5235)
This PR removes a few hundred thousand temporary seq allocations during
state transition - in particular, the flag seq was allocated per
validator while committees are computed per attestation.
2023-08-03 01:03:40 +02:00
henridf 28194468c9
Rename "data gas" to "blob gas" (#5216)
* Rename data gas to blob gas

* Update vendor/nim-eth and vendor/nim-web3
2023-08-02 22:07:57 +00:00
henridf 1da82a5dcd
Remove async blob queue from Request Manager (#5198)
* Remove async blob queue from Request Manager

* Review feedback

* Review feedback
2023-08-01 22:39:14 +02:00
tersec e9a17fb99f
remove workaround for long-since-fixed Nim bug (#5219) 2023-07-31 22:20:09 +00:00
tersec 909a7c2d07
use v1.4.0-beta.1 consensus spec tests (#5230) 2023-07-31 21:51:06 +00:00
Etan Kissling f8ba2e3882
generalize `ShufflingRef` acceleration logic (#5197) (#5204)
Split up the `ShufflingRef` acceleration logic into generically usable
parts and attester shuffling specific parts. The generic parts could be
used to accelerate other purposes, e.g., REST `/states/xxx/randao` API.
2023-07-31 18:12:15 +00:00
tersec 846e7c585b
Revert "Revert "generalize `ShufflingRef` acceleration logic (#5197)" (#5223)" (#5225)
This reverts commit 2ab4592a31.
2023-07-31 13:11:45 +00:00
tersec 2ab4592a31
Revert "generalize `ShufflingRef` acceleration logic (#5197)" (#5223)
This reverts commit eb3a30655b.
2023-07-31 08:05:32 +02:00
Philippe Schommers 74c993d616
feat: add support for Chiado (#5208) 2023-07-26 15:54:16 +03:00
Jacek Sieka e8379389e7
speed up state/block loading (#5207)
* speed up state/block loading

When loading blocks and states from db/era, we currently redundantly
check their CRC32 - for a state, this costs 50ms of loading time
presently (110mb uncompressed size) on a decent laptop.

* remove `maxDecompressedDbRecordSize` - not actually used on recent
data since we store the framed format - also, we're in luck: we blew
past the limit quite some time ago
* fix obsolete exception-based error checking
* avoid `zeroMem` when reading from era store

see https://github.com/status-im/nim-snappy/pull/22 for benchmarks

* bump snappy
2023-07-26 10:47:46 +03:00
Eugene Kabanov df80ae68fa
REST server metrics and connections API endpoints. (#5193)
* Add new REST endpoints to monitor REST server connections and new chronos metrics.

* Bump head versions of chronos and presto.

* Bump chronos with regression fix.

* Remove outdated tests which was supposed to test pipeline mode.

* Disable pipeline mode in resttest.

* Update copyright year.

* Upgrade test_signing_node to start use AsyncProcess instead of std library's osproc.
Bump chronos to check graceful shutdown.

* Update AllTests.

* Bump chronos.
2023-07-21 16:54:09 +02:00
Etan Kissling eb3a30655b
generalize `ShufflingRef` acceleration logic (#5197)
Split up the `ShufflingRef` acceleration logic into generically usable
parts and attester shuffling specific parts. The generic parts could be
used to accelerate other purposes, e.g., REST `/states/xxx/randao` API.
2023-07-20 10:25:39 +02:00
tersec 81c989660a
remove Bellatrix builder API support (#5162)
Co-authored-by: Etan Kissling <etan@status.im>
2023-07-20 04:37:54 +00:00
Etan Kissling da5bb96856
avoid Nim 2.0 issue when loading config (#5202)
To avoid Nim 2.0 issue https://github.com/nim-lang/Nim/issues/22284,
explicitly specify `ref SecondarySources` instead of using `auto`,
and add `{.raises.}` annotation
2023-07-19 20:26:30 +02:00
Zahary Karadjov 60f0a2f6a6
Version 23.7.0 2023-07-19 15:35:49 +03:00
Etan Kissling 971b4483c5
add `libnimbus_lc.a` C library (#5122)
Add a new C library for processing light client data based on the Nimbus
implementation. This can be used from other, non-Nimbus components.
2023-07-19 09:48:39 +02:00
Etan Kissling 346d05a95d
increase `getPayloadFromSingleEL` timeout (#5194)
When producing a local block and `runProposalForkchoiceUpdated` was
missed, `getPayloadFromSingleEL` adds additional ~500ms of latency.
Quick fix to avoid missing blocks to that.
2023-07-18 23:29:50 +03:00
Jacek Sieka b3b5238434
disable startup pruning (#5191)
it has been shown to cause long startup times - a better strategy is
needed
2023-07-18 23:29:23 +03:00
Etan Kissling dd35d2d309
delay `onSlotEnd` if there are duties (#5196)
We currently call `onSlotEnd` whenever all in-BN validator duties are
completed. VC validator duties are not awaited. When `onSlotEnd` is
processed close to the slot start, a VC may therefore miss duties.
Adding a delay before `onSlotEnd` improves this situation.
The logic can be optimized further if `ActionTracker` would track
`knownValidators` from REST separately from in-process ones.
2023-07-18 20:55:36 +02:00
Etan Kissling f98c33ad03
generalize `commonAncestor` function to `BlockId` (#5192)
To enable additional use cases, e.g., `/states/###/randao` beacon API,
`ShufflingRef` acceleration logic needs to be able to operate on parts
of the DAG that do not have `BlockRef`. Changing `commonAncestor` to
act on `BlockId` instead of `BlockRef` is a step toward that and also
simplifies the logic some more.
2023-07-18 17:37:53 +02:00
tersec 3a818ecb93
fall back to non-fcu fork choice on epoch boundaries (#5195)
* fall back to non-fcu fork choice on epoch boundaries

* Future[bool]

* fix

* Update beacon_chain/consensus_object_pools/consensus_manager.nim

Co-authored-by: Etan Kissling <etan@status.im>

* make things consistent with Opt[void] return

---------

Co-authored-by: Etan Kissling <etan@status.im>
2023-07-17 22:30:38 +02:00
Etan Kissling 2efc44a8ab
accelerate RANDAO computation for post-merge blocks (#5190)
Post-merge blocks contain all information to directly obtain RANDAO
without having to load any additional info. Take advantage of that to
further accelerate `ShufflingRef` computation. Note that it is still
necessary to verify that `blck` / `state` share a sufficiently recent
ancestor for the purpose of computing attester shufflings.

- new: 243.71s, 239.67s, 237.32s, 238.36s, 239.57s
- old: 251.33s, 234.29s, 249.28s, 237.03s, 236.78s
2023-07-15 22:16:56 +02:00
tersec 565edfa351
remove unused code (#5158)
* remove unused code

* add copyright header
2023-07-15 16:30:52 +00:00
tersec 703eead2e6
ensure the arrays in BeaconStateDiff are int-indexable in Nim 2.0 (#5189) 2023-07-13 23:25:10 +02:00
tersec 0be5b5b8f2
handle distinct nil more rigorously (#5180) 2023-07-12 17:33:19 +00:00
Etan Kissling 74bb4b1411
simplify RANDAO recovery in `ShufflingRef` acceleration (#5183)
Current RANDAO recovery logic is quite complex as it optimizes for the
minimum amount of database reads. Loading blocks isn't the bottleneck
though, so rather make the implementation more concise by avoiding the
complex strategy planning step. Note that this also prepares for an even
faster implementation for post-merge blocks in the future that extracts
RANDAO from `ExecutionPayload` directly if available, so even in cases
where efficiency is slightly lower, only historical data is affected.

`time nim c -r tests/test_blockchain_dag` (cached binary):

- new: 145.45s, 133.59s, 144.65s, 127.69s, 136.14s
- old: 149.15s, 150.84s, 135.77s, 137.49s, 133.89s
2023-07-12 17:27:05 +02:00
Jacek Sieka 0660ffcd3e
fix debug logging 2023-07-11 21:48:27 +02:00
henridf f13f23e21b
Complete Deneb bn side for /eth/v2/validator/blocks/{slot} (#5146)
* Complete Deneb bn side for /eth/v2/validator/blocks/{slot}

* Address review feedback

* More review feedback

* Tighten up when condition

* Update beacon_chain/rpc/rest_validator_api.nim

Co-authored-by: tersec <tersec@users.noreply.github.com>

---------

Co-authored-by: tersec <tersec@users.noreply.github.com>
2023-07-11 19:35:55 +00:00
Jacek Sieka a2adbf809f
Perform block pre-check before validating execution (#5169)
* Perform block pre-check before validating execution

When syncing, blocks have not been gossip-validated and are therefore
prone to trivial faults like being known-unviable, duplicate or missing
their parent.

In addition, the duplicate-block check in BlockProcessor was not
considering the quarantine flow and would therefore cause
recently-quarantined blocks to be silenty dropped when their parent
appears delaying the sync end-game and thus causing longer startup
resync time.

This PR verifies trivial conditions before performing execution
validation thus avoiding duplicates and missing parents alike.

It also ensures that the fast-sync EL mode is used for finalized blocks
even if the EL is timing out / slow to respond - this allows the CL to
complete its sync faster and switch to "normal" lock-step at the head of
the chain more quickly, thus also allowing the EL to access the latest
consensensus information earlier.

* oops

* remove unused constant
2023-07-11 18:55:51 +02:00
Jacek Sieka ca1775f725
Fetch by-root request directly from quarantine (#5167)
When the requestmanager is busy fetching blocks, the queue might get
filled with multiple entries of the same root - since there is no
deduplication, requests containing the same root multiple times will be
sent out.

Also, because the items sit in the queue for a long time potentially,
the request might be stale by the time that the manager is ready with
the previous request.

This PR removes the queue and directly fetches the blocks to download
from the quarantine which solves both problems (the quarantine already
de-duplicates and is clean of stale information).

Removing the queue for blobs is left for a future PR.

Co-authored-by: tersec <tersec@users.noreply.github.com>
2023-07-11 18:22:02 +02:00
tersec 174c33e5fa
residual cleanup from https://github.com/status-im/nimbus-eth2/pull/5152 (#5181) 2023-07-11 14:36:37 +00:00
Jacek Sieka 185684daf8
simplify deposit syncing message (#5177)
and make it less alarming since there's no action required from the user
typically
2023-07-11 08:01:56 +02:00
Jacek Sieka 1bac029935
avoid unnecessary buffer zero:ing (#5178) 2023-07-11 08:01:45 +02:00
tersec a1ddd447bc
use correct import name for kzg4844 (#5179) 2023-07-10 23:55:40 +00:00
Etan Kissling 5115aaedb7
early exit `commonAncestor` when comparing with `finalizedHead` (#5174)
* early exit `commonAncestor` when comparing with `finalizedHead`

As all `BlockRef` lead to `finalizedHead` (`parent == nil`),
can shortcut in that situation and immediately return `finalizedHead`
if passed as one of the arguments.

* typo in comment

* add test from #5152

Co-authored-by: tersec <tersec@users.noreply.github.com>

* add note about test complexity

* regenerate test summary

---------

Co-authored-by: tersec <tersec@users.noreply.github.com>
2023-07-10 20:36:25 +00:00
henridf 86e33b0a3f
Remove stale denebImplementationMissing (#5173)
This should have been removed as part of #5137
2023-07-10 17:17:33 +02:00
henridf 23edc4f960
Finish readValue for deneb RestPublishedBeaconBlockBody (#5172)
(Stray piece that got left behind a previous PR...)
2023-07-10 14:57:11 +00:00
tersec 49056c3e39
use v1.3.0 stability subnet behavior when appropriate (#5156) 2023-07-07 04:16:12 +00:00
Jacek Sieka 862897124a
rest: allow state root lookups up to 8k slots from head (#5155)
* rest: allow state root lookups up to 8k slots from head

* avoid int, clean up error messages
2023-07-06 12:53:35 +02:00
Eugene Kabanov ac1b02698a
VC: Use scoring function to select best attestation data when using multiple BNs. (#5101)
* Initial commit.

* Move score selection log statement to debug level.

* Fix proper float64 log format.

* Cleanup imports and legacy code.

* Address review comments.

* Address review comments.

* Fix scoring function.

* Address review comments.

* Address review comments 2.
Fix registerBlock post-rebase issues.

* Simplify innerLoop decision making.

* Make getAttestationDataScore() more testable.
Add tests for getAttestationDataScore().

* Add modified AllTests copy.
2023-07-06 11:14:22 +02:00
tersec 1e2d3b12c4
update some consensus-specs ref URLs to v1.4.0-beta.0 (#5163) 2023-07-05 16:02:55 +00:00
tersec 583eb4db17
work around some CastSizes warnings and fix unreachable code warnings (#5149) 2023-06-30 21:34:04 +02:00
tersec 0be95571a7
update some consensus spec URLs to v1.4.0-beta.0 (#5150) 2023-06-30 16:12:54 +00:00
tersec b4c4f0955e
https://github.com/ethereum/consensus-specs/pull/3421 https://github.com/ethereum/execution-apis/pull/420 (#5147) 2023-06-30 08:14:20 +00:00
henridf 99264d7507
Local validator proposals (#5137)
* Implement local validator block proposals

* Address review feedback

* Fix rebase issue
2023-06-30 07:39:41 +00:00
tersec ba94dc849f
opt-out support for v1.4.0 attestation stability subnets (#5128)
* opt-out support for v1.4.0 attestation stability subnets

* document debug-prefix-flag-policy and remove temporary from flag name
2023-06-30 05:23:15 +00:00
tersec d5a6d51ede
Web3Signer support only for only Bellatrix and later (#5107)
Co-authored-by: Etan Kissling <etan@status.im>
2023-06-29 09:36:43 +00:00
tersec dc5687093f
update gossip validation for v1.4.0-beta.0 (#5133)
Co-authored-by: Etan Kissling <etan@status.im>
2023-06-29 08:34:21 +00:00
henridf 7b0e3b1c04
Review feedbac from #5106 (#5141) 2023-06-29 01:17:54 +00:00
henridf 1234900065
Add blob handling to message router (#5106)
* Add blob handling to message router

* address review feedback

* Fix typos
2023-06-28 17:55:31 +00:00
Etan Kissling 41b93ae57a
fix code for Nim `v1.6.14` compatibility (#5131)
With `v1.6.14` there is compilation issue in `trusted_node_sync` where
a type is not inferred automatically anymore for a `nil` instance.
Fix it so we can bump the compiler.

See https://github.com/status-im/nimbus-build-system/pull/63
2023-06-28 17:21:43 +00:00
Eugene Kabanov 5e1a0eac85
VC: Runtime config (#5130)
* Make VC able to understand any type of `/eth/v1/config/spec` response without any changes in source code.
Update compatibility checking.
Now VC is able to obtain any constant from `spec` call.

* Remove RestSpecVC declaration.
2023-06-28 15:33:38 +02:00
Eugene Kabanov c534a285b9
Fix polling block monitor from deadlocks when connection has been lost. (#5109)
Fix events block monitor from losing connection forever.
Add node to block received log statement.
2023-06-28 15:33:21 +02:00
Eugene Kabanov c2c5d80a4f
VC: Remote BN clock offset monitoring. (#4846)
* Initial commit.

* Add algorithm in comment.
Remove delays.
Fix logging statement issues.
Change update from epoch to slot.

* Obtain timestamp earlier.

* Add processing delays into algorithm.

* Fix time offset logging to produce integers instead of strings.

* Address review comments.

* Fix copyright year.
Fix updateStatus().

* Remove fields from Slot start log statement.
Fix issues when BN do not support Nimbus Extensions.
Rename metric name and type change.

* Add beacon role to disable time offset check manually.
2023-06-28 15:33:07 +02:00
tersec bf575aac57
make OnFooBlockAdded callback types generic (#5134) 2023-06-28 13:30:49 +00:00
Zahary Karadjov 970f5dfc4e
Merge branch 'stable' into unstable 2023-06-27 05:40:58 +03:00
Zahary Karadjov 187e1a0633
Version 23.6.1 2023-06-27 03:58:14 +03:00
Jacek Sieka 17e74c0c8b
don't check legacy tables for pruning (#5116)
These tables can't be deleted from (read-only) and would be too slow to
delete from anyway due to the inefficient storage format in use.

* slow down startup clearing too
* remove unused del function
2023-06-26 19:30:45 +03:00
henridf ac3d8939a8
Add Deneb handling to state_transition.makeBeaconBlock() (#5118)
* Add Deneb handling to state_transition.makeBeaconBlock()

* review feedback
2023-06-26 16:05:02 +00:00
tersec a2bdc2d723
revert 2023-06-26 08:38:05 +00:00
tersec dbd7611983
not a real change 2023-06-26 08:36:39 +00:00
tersec ba597ef0a2
per-validator payload builder configuration (#5062) 2023-06-25 12:00:17 +00:00
tersec 614202e30d
automated consensus spec URL updating to v1.4.0-beta.0 (#5121) 2023-06-24 15:43:30 +00:00
Jacek Sieka d9bff54d57
don't check legacy tables for pruning (#5116)
These tables can't be deleted from (read-only) and would be too slow to
delete from anyway due to the inefficient storage format in use.

* slow down startup clearing too
* remove unused del function
2023-06-24 16:14:28 +02:00
tersec 9dbf63d8ca
use v1.4.0-beta.0 consensus spec test vectors (#5120) 2023-06-24 12:27:15 +00:00
tersec b62664915e
consensus spec v1.4.0 attestation stability subnets (#5092) 2023-06-23 09:30:46 +00:00
henridf fa212515f5
Implement validor client proposals for Deneb (#5094) 2023-06-23 09:30:16 +00:00
Tanguy 11d18acfeb
Add Yamux experimental support (#5080) 2023-06-23 09:16:30 +02:00
Etan Kissling 8a853fc4e4
extract LC sync task scheduling helper function (#5117)
The helper function to compute delay until next light client sync task
can be useful from more general purpose contexts. Move to helpers, and
change it to return `Duration` instead of `BeaconTime` for flexibility.
2023-06-23 06:19:51 +00:00
Etan Kissling cbc4cef981
bump `gnosis-chain-configs` to `19060cb894ea0fb8ff3c8ec2fd26c0433f56c52c` (#5114)
https://github.com/gnosischain/configs/pull/12

- Schedule shapella for gnosis
2023-06-22 11:14:02 +03:00
Etan Kissling bab6a77abc
optimize LC sync task scheduling (#5112)
- When syncing `LightClientUpdatesByRange`, and peer replies with
  fewer periods than requested, no need to delay next request.
- When `FinalityUpdate` / `OptimisticUpdate` sync fails,
  no need to retry immediately.
2023-06-21 17:44:49 +00:00
Etan Kissling 6ec6b99ef6
extract `nextLightClientSyncTask` helper (#5108)
The `nextLightClientSyncTask` helper is generic enough to be useful for
contexts besides libp2p based light client sync. Extract it.
2023-06-21 06:28:14 +00:00
Etan Kissling d670af5bf5
align style when calling LC libp2p API (#5104)
The `UpdatesByRange` API takes `startPeriod / count`, but is internally
called by `Slice`. Move the logic that converts from the `Slice` to the
caller to reduce complexity inside the used `doRequest` function.
2023-06-20 20:21:32 +00:00
Eugene Kabanov e2f7065432
Fix iteration issue. (#5105)
Add elapsed_time to log statement.
2023-06-20 16:41:20 +00:00
Zahary Karadjov 6b683924ce
Version 23.6.0 2023-06-20 02:11:44 +03:00
Zahary Karadjov b32fac0a25
Merge branch 'stable' into unstable 2023-06-20 02:09:05 +03:00
Zahary Karadjov 503cde366c
Switch to --history:prune by default 2023-06-20 02:08:48 +03:00
Etan Kissling 2722778ce5
reduce `nim-eth` dependencies just for RNG (#5099)
We have several modules that import `nim-eth` for the sole purpose of
its `keys.newRng` function. This function is meanwhile a simple wrapper
around `nim-bearssl`'s `HmacDrbgContext.new()`, so the import doesn't
really serve a use anymore. Replace `keys.newRng` with the direct call
to reduce `nim-eth` imports.
2023-06-19 22:43:50 +00:00
henridf 0f8866d672
Add Deneb beacon API types (#5060)
* Add Deneb beacon API types

- Introduce the `ProduceBlockResponseV2` type for handling responses to `GET /eth/v2/validator/blocks/{slot}` (https://ethereum.github.io/beacon-APIs/?urls.primaryName=dev#/Validator/produceBlockV2)

- Introduce the `RestPublishedSignedBlockContents` type for handling the request body to `POST /eth/v1/beacon/blocks` (https://ethereum.github.io/beacon-APIs/?urls.primaryName=dev#/ValidatorRequiredApi/publishBlock)

And a few other small changes towards Deneb block production support.

* Address review feedback

* Post-rebase fixes
2023-06-19 08:56:52 +00:00
Etan Kissling 3a6b50cf21
require sync committee supermajority in CI (#5085)
* require sync committee supermajority in CI

To better catch problems with sync committee messages in CI, extend
local testnet simulation to also verify that each block is signed
by a supermajority of the sync committee.

Requires #5083 and #5084

* lint
2023-06-17 06:00:01 +00:00
Etan Kissling 8c6c8a0ffa
use correct slot when producing sync aggregate around forks (#5089)
`produceSyncAggregate` is called in new slot when block is produced,
while the other functions in `sync_committee_msg_pool` are called in
previous slot. So, need to subtract 1 slot when producing sync aggregate
to accept the signatures using the old digest during fork transition.
2023-06-16 21:30:36 +00:00
Etan Kissling aa54760391
avoid implicit `CaseTransition` in keystore Json parsing (#5061)
Override default Json parsing for keystore case objects to avoid
`CaseTransition` logic to be emitted. When parsing the discriminator,
reinitialize the entire object instead of implicitly changing it,
to avoid UB and also possible oversights when extending the object.
See https://github.com/status-im/nim-serialization/pull/59
2023-06-16 19:02:53 +02:00
tersec 591c2246d5
update consensus spec URLs to v1.4.0-alpha.3 (#5088) 2023-06-16 16:45:09 +00:00
henridf ae29babd9f
Implement blob sidecar signatures (#5072)
* Implement blob sidecar signatures

* Update allTests
2023-06-16 15:47:18 +00:00
Etan Kissling 8eec6ab221
avoid `ProveField` warning in `network_metadata` (#5066)
Before assigning to `genesisData` or returning `cfg`, have to check that
metadata is not `incompatible` to avoid `ProveField` warning.
The way how we use it was already correct (`incompatible` unreachable).
Use `case` syntax to silence the warning, and add comments referring to
the existing checks that make `incompatible` unreachable.
2023-06-16 12:15:42 +00:00
Etan Kissling e9262ab6a3
ensure sync duties for next epoch are registered in time (#5084)
* ensure sync duties for next epoch are registered in time

For attestations, VC queries duties for current and next epoch.
For sync messages, VC queries for current and next period (if soon).
This means that for sync messages we don't actually have the duties for
next epoch in all situations, leading to situation where VC may miss
sync duties in the final slot of an epoch when using. As duties remain
same within a sync committee period, simply copy them over to next epoch
to avoid this situation without adding network latency.

* Update beacon_chain/validator_client/duties_service.nim

Co-authored-by: Jacek Sieka <jacek@status.im>

---------

Co-authored-by: Jacek Sieka <jacek@status.im>
2023-06-16 08:42:57 +00:00
Jacob Shufro 11f5250ce4
Lower ClientMaximumValidatorIds to match beacon spec limit (#5082) 2023-06-16 09:01:16 +02:00
Etan Kissling 6f68c1a46e
query sync duties for correct slot (#5083)
VC currently misses sync committee duties for first slot of most epochs
because the 1 slot offset is not taken into account. Duties for the next
slot must be used during any given current slot. We use the correct slot
for processing the duty, but do not use the correct slot for fetching.
2023-06-16 06:33:13 +00:00
Etan Kissling a6f0a7a55d
broaden `SszError` catches to `SerializationError` (#5081)
`nim-serialization` is tagged with `{.raises:[SerializationError].}` so
it is no longer sufficient to catch `SszError` in some situations.
`SszError` inherits from `SerializationError`, so broadening the caught
exception types can be done now, to enable bumping `nim-serialization`.

https://github.com/status-im/nimbus-eth2/pull/5043#issuecomment-1584227993

#5061 is also needed to bump `nim-serialization`.
2023-06-15 16:29:13 +00:00
Tanguy 98ab0afcc3
Bump libp2p (#5079)
* Bump libp2p

* Switch to unstable
2023-06-15 12:59:35 +00:00
Etan Kissling 142a3e91e5
`ProveField` cleanups in `keystore` (#5050)
Cleanup for `ProveField` warnings in `keystore` module.
Note that `ProveField` is disabled by default in makefile, but sometimes
these pop up when doing a regular `nim c`, and cleaning these may allow
enabling the warning in some future.
2023-06-15 14:53:42 +02:00
tersec ce1503b51f
remove web3signer block signing V1 (phase0 only) support (#5014) 2023-06-15 14:52:19 +02:00
tersec 939b3002e4
remove removed getDebugChainHeadsV1; fix API links in documentations (#5058) 2023-06-15 09:58:38 +00:00
Etan Kissling 411d809c82
split file loading from parsing in helpers (#5054)
* split file loading from parsing in helpers

In `readSszForkedHashedBeaconState` and `readRuntimeConfig`, split the
part that loads the file from the part that parses the file. The parsing
portion can be reused with that, e.g., when loading from the network.

* add missing export marker
2023-06-14 19:48:50 +00:00
Etan Kissling 114453e67d
remove `SAFE_SLOTS_TO_UPDATE_JUSTIFIED` (#5078)
The `SAFE_SLOTS_TO_UPDATE_JUSTIFIED` constant is no longer used in any
EF or Gnosis network metadata definitions. We can remove it.

Upstream changes:

- https://github.com/eth-clients/eth2-networks/pull/87
- https://github.com/gnosischain/configs/pull/10

Which we updated here:

- https://github.com/status-im/nimbus-eth2/pull/5071
- https://github.com/status-im/nimbus-eth2/pull/5077
2023-06-14 18:18:53 +00:00
tersec c27a65129b
add ability to load KZG trusted setup via runtime flag (#5075) 2023-06-14 10:52:00 +02:00
Eugene Kabanov 927180f36f
VC+BN: Validator voluntary exits through the Keymanager API (#5020)
* Initial commit.

* Address review comments.
2023-06-14 08:46:01 +02:00
Eugene Kabanov c0e5c26da1
VC: Use SSZ encoding while processing blocks data (#4999)
* Refactor api.nim to provide more informative failure reasons.
Distinct between unexpected data and unexpected code.
Deprecate Option[T] usage.

* Fix 400 for produceBlindedBlock().
Get proper string conversion for strategy.

* Fix SSZ encoded versions of ProduceBlockResponseV2, ProduceBlockResponseV2 can be received and decoded.
Fix done() warnings.
Bump presto.

* Fix compilation error with new presto.
Use TcpNoDelay option for Web3Signer.

* Fix produceBlockV2() should provide SSZ responses too.

* Address block encoding issue.

* Fix signing test.

* Bump presto.

* Address review comments.
2023-06-14 08:04:15 +02:00
Etan Kissling 6671965fd9
split SSZ and JSON parsing logic in LC handler (#5055)
Use separate functions per format when parsing LC data from REST.
This allows to process events from the eventstream more directly,
as they are always JSON not SSZ. And also makes the code cleaner.
2023-06-13 21:28:17 +00:00
tersec 54cc7bb7a1
update some consensus-spec URLs to v1.4.0-alpha.3 (#5074) 2023-06-13 21:07:39 +00:00
tersec 9277326c98
fix misspelling in log message (#5069) 2023-06-13 17:08:41 +00:00
Etan Kissling 20d6b4751e
avoid `ProveField` warning in `keyGen` (#5067)
Assigning to fields of `var` case objects emits `ProveField` warnings.
We disable them in `make` based builds but they may pop up when building
manually with `nim c`. Suppress them for the `keyGen` function, as we
assign to `result.value` separately from `result.ok` to avoid copying.
2023-06-13 16:45:00 +00:00
tersec 788cdb7133
automated v1.4.0-alpha.2 to v1.4.0-alpha.3 consensus spec URL updates (#5065) 2023-06-13 14:03:49 +00:00
tersec d26058b929
use v1.4.0-alpha.3 consensus spec test vectors (#5063) 2023-06-13 10:37:10 +00:00
tersec d7b546fab5
fix UnusedImport warning (#5064) 2023-06-13 10:03:20 +00:00
Etan Kissling a3229a6b9b
extract helper for validating `UpdatesByRange` consistency (#5053)
Reduce code duplication when checking response of `UpdatesByRange`.
2023-06-12 14:22:32 +02:00
Etan Kissling 40415ab86c
`ProveField` cleanups in `forks` (#5049)
* `ProveField` cleanups in `forks`

Some more cleanup for `ProveField` warnings in `forks` module.
Note that `ProveField` is disabled by default in makefile, but sometimes
these pop up when doing a regular `nim c`, and cleaning these may allow
enabling the warning in some future.

* use syntax that works if passed to multiple args of call
2023-06-12 12:03:59 +00:00
tersec 22208836b1
automated v1.4.0-alpha.1 to v1.4.0-alpha.2 consensus spec URL updates (#5056) 2023-06-10 09:56:54 +00:00
tersec 1b77831758
Revert "remove removed Debug/getDebugChainHeads V1 (#4970)" (#5057)
This reverts commit 6ebcbbdf4e.
2023-06-10 08:08:01 +00:00
tersec 6ebcbbdf4e
remove removed Debug/getDebugChainHeads V1 (#4970)
* remove removed Debug/getDebugChainHeads V1

* REST API tests
2023-06-10 08:07:36 +00:00
tersec 9e14d904ac
https://github.com/ethereum/consensus-specs/pull/3359 (#5047) 2023-06-10 05:39:10 +00:00
tersec 6a9ed6ddd5
use v1.4.0-alpha.2 consensus spec test vectors (#5051) 2023-06-10 01:45:53 +00:00
tersec 54cb06ea06
fix chronos deprecation warning (#5048) 2023-06-09 23:43:28 +00:00
Etan Kissling a0d63c50ee
do not use untrusted `genesis.ssz` with `--trusted-block-root` (#5025)
When using trusted node sync with `--trusted-block-root`, the remote
server is only trusted for data availability, not for correctness.
As a downloaded genesis state cannot be validated for correctness,
require it to be passed via the network metadata `genesis.ssz` file
for `--trusted-block-root` mode. Network metadata is considered trusted
as it is provided by the user and not by the remote server.
Further adds a check for consistent `genesis_time` when using `StateId`
based trusted node sync. This is just a sanity check to avoid spreading
blatantly incorrect data, similar to existing `genesis_validators_root`
checks.
2023-06-09 22:11:14 +02:00
henridf eda631cd7a
Blob accounting fixes/tweaks (#5044)
- Replace asset by warning
- change inequality to lt
2023-06-09 10:52:03 +00:00
Tanguy 46a12639b8
Read messages before applying quota to avoid mplex backpressure issues (#4697)
* Apply global quota after reading messages

* Also wait quota for failed requests

* Better integration

* comments
2023-06-08 14:20:41 +00:00
Jacek Sieka 8db87a0cfc
bump chronos (#5039)
* separate futures import / Defect cleanups
* unittest2 fix to allow `stackTrace` as a symbol name generally
2023-06-08 13:42:19 +00:00
Eugene Kabanov effe8b7f90
VC: Remote BN received block monitoring. (#4856)
* Initial commit with both methods enabled: `poll` and `event`.

* Address review comments.

* Address review comments.
Fix copyright years.

* After bump fixes.
2023-06-08 10:44:32 +02:00
tersec e8c6af0636
add getNextWithdrawals Beacon API (#5021) 2023-06-07 21:27:15 +00:00
henridf 3bda24d4d1
Remove unused param from prepareForkedBlockReading (#5037) 2023-06-07 10:39:04 +00:00
tersec 30c859e8a4
update some consensus-spec URLs to v1.4.0-alpha.1 (#5035) 2023-06-06 17:09:48 +00:00
tersec 58e3de3bf2
add REST call signatures for getStateRandao and getNextWithdrawals (#5033) 2023-06-06 10:01:05 +00:00
Jacek Sieka fc833d40f6
bump chronos (#5032)
* ratelimiter crash fix
* implicit returns now available!
2023-06-05 18:52:16 +00:00
henridf c2aa8313c3
Sync: fix backfill blob iteration order (#4993)
* Sync: fix backfill blob iteration order

* Address review feedback

* Add comment clarifying reason for func top-level placement
2023-06-05 14:42:27 +00:00
jangko a24f3cde7e
more flexible compile time resource path 2023-06-05 12:52:25 +07:00
Etan Kissling d99caf1ad9
fix `Access-Control-Allow-Origin` response (#5028)
Since #3976, CORS functionality is broken. Fix it to work again:

- Use `--rest-allowed-origin` instead of `--keymanager-allowed-origin`
  to specify CORS `Access-Control-Allow-Origin` header for beacon-APIs.

- Actually pass CORS config to `nim-presto` once more.
2023-06-04 16:46:00 +00:00
Etan Kissling 336acbd39b
display `ExecutionAddress` as hex string (#5029)
When logging `ExecutionAddress`, serialize it as a hex string instead of
as a byte array.
2023-06-04 16:15:14 +00:00
Etan Kissling 12381d5f33
also pack attestations where LMD vote is orphaned (#5002)
* also pack attestations where LMD vote is orphaned

When `attestation.data.beacon_block_root` gets orphaned, attestations
with a good `attestation.data.target.root` may still be valuable.
The LMD GHOST vote is not relevant for attestation rewards.

Switch to use the FFG vote (`attestation.data.target.root`) instead,
gossip validation ensures it is an ancestor of `beacon_block_root`.

* lint
2023-06-04 10:39:12 +00:00
tersec f86febc111
update consensus spec URLs to v1.4.0-alpha.1 (#5027) 2023-06-04 10:32:50 +00:00
tersec b25ca0833b
use v1.4.0-alpha.1 consensus spec test vectors (#5026) 2023-06-03 21:55:08 +00:00
tersec e6a5f03717
add comment to state diffs explaining eth1 vote tracking (#5023) 2023-06-03 17:48:19 +00:00
tersec ee71b6cc36
update consensus spec URLs to v1.4.0-alpha.0 (#5022) 2023-06-02 12:59:38 +00:00
tersec e8e67ec771
allow payload builder client to be function of validator/proposer (#5015)
* allow payload builder client to be function of validator/proposer

* fileExists has side effects on Windows and only Windows

* another not-always-func
2023-06-02 11:06:33 +00:00
Etan Kissling 005a35597f
handle one of the `ProveField` warnings (#5013)
* handle one of the `ProveField` warnings

When assigning between `ForkyHashedBeaconState`, suppress `ProveField`
warning, as `tgt.kind == src.kind` was already checked, but compiler
doesn't understand that (as we only `case tgt.kind`).

* Update beacon_chain/spec/forks.nim

* Update beacon_chain/spec/forks.nim
2023-06-02 01:25:49 +00:00
tersec 7a7573d7d0
fix chronos done -> completed deprecation warnings (#5016) 2023-06-01 10:04:30 +02:00
Etan Kissling 1086909e0b
ensure `quit` on config error with `IOError` (#5011)
When there is an `IOError` while logging a configuration error,
we don't actually `quit` the program. Catch `IOError` to always `quit`.
2023-05-31 20:21:49 +00:00
Etan Kissling 6cd63a89d8
annotate `secondarySources` with `{.raises.}` (#5012)
`sources.addConfigFile` may raise `ConfigurationError`, annotate caller
to propagate that error explicitly.
2023-05-31 19:28:33 +00:00
Etan Kissling 73b3f40e8d
use correct exception in `parseCmdArg(enr.Record)` (#5009)
* use correct exception in `parseCmdArg(enr.Record)`

`parseCmdArg` is expected to raise `ValueError` but for `enr.Record` we
currently raise `ConfigurationError`. Change to `ValueError` instead.

* lint
2023-05-31 19:07:06 +00:00
tersec bc458921ec
don't require optional fields importing slashing protection information (#4997) 2023-05-31 18:51:00 +03:00
tersec c9f1bf21d6
refactor state diffs not to require two states in memory (#4986) 2023-05-30 11:55:32 +03:00
Eugene Kabanov 528d082fc0
VC: Logging changes (#4994)
* Refactor api.nim to provide more informative failure reasons.
Distinct between unexpected data and unexpected code.
Deprecate Option[T] usage.

* Fix generated reason to not include opt[t].

* Fix 400 for produceBlindedBlock().
Get proper string conversion for strategy.

* Bump copyright years.
2023-05-30 11:38:34 +03:00
Etan Kissling 750722dd75
advance FC time if block/attestation is early (#4992)
When processing blocks/attestations that are slightly early, within the
spec allowed `MAXIMUM_GOSSIP_CLOCK_DISPARITY`, bump FC time accordingly.
2023-05-26 08:03:49 +00:00
tersec 1f535336da
automated v1.3.0 to v1.4.0-alpha.0 consensus spec URL updates (#4996)
* automated v1.3.0 to v1.4.0-alpha.0 consensus spec URL updates

* add copyright year headers
2023-05-26 00:14:28 +02:00
tersec d1941b670a
refactor payload builder REST client usage (#4973)
* refactor payload builder REST client usage

* change HTTP response code
2023-05-25 18:38:56 +03:00
Etan Kissling 00728e9bb7
use `executionValid` bit in `BlockRef` (#4956)
Update `beacon_node/rpc` usage for the new `executionValid` field.
2023-05-25 15:57:24 +02:00
tersec 10569ff2e9
use v1.4.0-alpha.0 consensus spec test vectors (#4990) 2023-05-25 07:55:00 +00:00
Etan Kissling f16c368f5c
allow `NeverTested` > `Working` transition for EL connection (#4991)
Since #4960, the EL connection status can no longer transition from
`NeverTested` to `Working`. Fix that, and also consider `NeverTested`
connections as online for the purpose of the `el_offline` REST response.
2023-05-25 10:39:47 +03:00
Jacek Sieka 74b670a4c9
reenable incbin on 64-bit platforms (#4988)
turns out we're still making releases for 32-bit platforms 😲
2023-05-24 23:43:41 +03:00
Jacek Sieka 8db65ef574
check flag before slower hash comparison 2023-05-23 09:09:22 +02:00
tersec 7577c625b3
implicitly backfill pre-finalized block execution validity from finalized block (#4984) 2023-05-23 08:26:31 +02:00
henridf 5ef748b19d
Clarify addOrphan error/logging (#4981)
* Clarify addOrphan error/logging

addOrphan returned a bool to indicate success. Change this to a Result
so that different errors can be distinguished.

* Update beacon_chain/consensus_object_pools/block_quarantine.nim

Co-authored-by: tersec <tersec@users.noreply.github.com>

* Update beacon_chain/gossip_processing/gossip_validation.nim

---------

Co-authored-by: tersec <tersec@users.noreply.github.com>
2023-05-21 17:47:00 +00:00
tersec cd087b9a43
replace `optimisticRoots` table with field in `BlockRef` (#4969)
* replace optimisticRoots table with field in BlockRef

* copyright year

* mark finalized blocks as verified on load

* Update beacon_chain/consensus_object_pools/block_dag.nim

Co-authored-by: Etan Kissling <etan@status.im>

* expand non-optimistic block checking to all pre-merge blocks; refactor markBlockVerified to use BlockRef rather than block root and remove superfluous caller in newPayload path replaced by addResolvedHeadBlock BlockRef construction

* don't treat finalized block specially; VALID status is sticky

---------

Co-authored-by: Etan Kissling <etan@status.im>
2023-05-20 12:18:51 +00:00
henridf 0044c18c01
Add reqCount to BlobSidecar range request done log (#4982) 2023-05-19 20:14:41 +00:00
henridf 7fb9a51c76
Fix blobkey calculation (#4978) 2023-05-19 16:51:13 +00:00
henridf 1cf777c64b
Fix sync for blocks older than MIN_EPOCHS_FOR_BLOB_SIDECARS_REQUESTS (#4977)
When doing sync for blocks older than
MIN_EPOCHS_FOR_BLOB_SIDECARS_REQUESTS, we skip the blobs by range
request, but we then pass en empty blob sequence to
validation, which then fails.

To fix this: Use an Option[Blobsidecars] to allow expressing the
distinction between "empty blob sequence" and "blobs unavailable". Use
the latter for "old" blocks, and don't attempt to run blob validation.
2023-05-19 16:25:11 +00:00
henridf 4b23c4292d
Remove unused proc getBlobSidecar (#4979) 2023-05-19 16:01:07 +00:00
Jacek Sieka c14d396718
harden req/resp peer scoring (#4966) 2023-05-19 15:01:27 +03:00
Zahary Karadjov a6b3e02ce0
Merge branch 'unstable' into dev/etan/rd-shufflingacc 2023-05-19 14:46:23 +03:00
Zahary Karadjov 4c3850f7df
Disable the use of incbin as it breaks the retail build on Linux/ARM 2023-05-19 04:08:02 +03:00
cheatfate a36cacda44
New stricter beacon-node URL parsing 2023-05-19 02:02:20 +03:00
Zahary Karadjov a167424fc0
Version 23.5.1 2023-05-18 20:10:12 +03:00
henridf 01549f6aa4
Wire in blob validation (#4864)
* Wire in blob validation

* Remove unused "is_data_available"

* Log blobs when blob validation fails
2023-05-17 13:55:50 +00:00
Etan Kissling c70fd8fe97
Merge branch 'unstable' into dev/etan/rd-shufflingacc 2023-05-17 14:06:31 +02:00
tersec 74511f61d1
Use withdrawal credentials as default fee recipient (#4968) 2023-05-17 07:56:37 +03:00
Etan Kissling 40e89937c5
segregate sync committee messages by period / fork (#4953)
`SyncCommitteeMsgPool` grouped messages by their `beacon_block_root`.
This is problematic around sync committee period boundaries and forks.
Around sync committee period boundaries, members from both the current
and next sync committee may sign the same `beacon_block_root`; mixing
the signatures from both committees together is a mistake. Likewise,
around fork transitions, the `signing_root` changes, so those messages
also need to be segregated.
2023-05-17 07:55:55 +03:00
zah e8572c0246
More relaxed parsing of RemoteKeystores to regain compatibility with version 1 (#4967) 2023-05-16 19:06:16 +03:00
tersec 5aeb5d9209
increase builder API getHeader timeout (#4964) 2023-05-16 16:41:55 +03:00
Jacek Sieka 83393cea8d
dependent slot helpers 2023-05-16 11:04:25 +02:00
Jacek Sieka 3acbb3ea6c
fix missing shuffling dependent slot computation 2023-05-16 10:05:35 +02:00
Etan Kissling adcabf9ad7
cleanup gossip message handler setup (#4961)
* cleanup gossip message handler setup

Reduce duplication and make gossip handler setup easier to read.

* explicit returns for async validators
2023-05-16 09:46:41 +02:00
Etan Kissling 3a92bf3914
use dependent root as `execution_optimistic` basis for duties (#4955)
The validator beacon APIs `getAttesterDuties`, `getProposerDuties`, and
`getSyncCommitteeDuties`, have reported the `execution_optimistic`
state for the current head block. This can lead to a race if duties are
requested around the slot start, if a new head block is currently being
processed by the EL, during which the BN head may be briefly optimistic.

`execution_optimistic` is documented in beacon APIs as:

> True if the response references an unverified execution payload.
> Optimistic information may be invalidated at a later time.
> If the field is not present, assume the False value.

As the duty endpoints reference the shuffling dependent root instead of
the currently selected head block, `execution_optimistic` is now fetched
based on that shuffling dependent block root. As this dependent block is
in the past it doesn't usually become optimistic when adding new blocks.

Note that the endpoints requested 4/8 seconds into the slot that perform
the actual duties instead of just querying for duty schedule, still
report `execution_optimistic` based on the BN head block.
2023-05-15 23:42:42 +03:00
zah 9b9c58c507
Don't report very brief EL connection interruptions on user-visible log levels (#4960) 2023-05-15 23:40:47 +03:00
Etan Kissling dbba003a38
Revert "Revert "accelerate `getShufflingRef` (#4911)" (#4958)"
This reverts commit 748be8b67b.
2023-05-15 17:41:40 +02:00
Etan Kissling 748be8b67b
Revert "accelerate `getShufflingRef` (#4911)" (#4958)
This reverts commit ea97e93e74.
2023-05-15 15:25:51 +00:00
henridf 573228ffa0
Rename eth1/ -> el/ and eth1_monitor.nim -> el_monitor.nim (#4944) 2023-05-15 05:05:12 +00:00
Etan Kissling 98c30f600b
fix `getSyncCommitteeDuties` for `ALTAIR_FORK_EPOCH`.period (#4954)
When fetching historical `getSyncCommitteeDuties` for the very first
sync committee period, the case must be handled where Altair may not
have been scheduled on a sync committee period boundary.
2023-05-14 23:56:50 +00:00
tersec 6024b3e508
rm unused function and unexport some others (#4941) 2023-05-14 22:56:14 +00:00
Etan Kissling deb5818587
use compile-time year in `--version` copyright (#4946)
Instead of using current runtime year, use compile-time year in notice.

```
nimbus_beacon_node --version
```
2023-05-13 04:30:49 +00:00
Etan Kissling 0701038f76
log `bls_to_execution_changes_len` in `shortLog` (#4949)
* log `bls_to_execution_changes_len` in `shortLog`

When logging blocks to console, include num BLS to Execution changes.

* fix

* lint
2023-05-13 04:29:46 +00:00
tersec a9f080d92d
small validator pool proc/func and unused symbol cleanup (#4945) 2023-05-12 19:21:43 +00:00
Etan Kissling ea97e93e74
accelerate `getShufflingRef` (#4911)
When an uncached `ShufflingRef` is requested, we currently replay state
which can take several seconds. Acceleration is possible by:

1. Start from any state with locked-in `get_active_validator_indices`.
   Any blocks / slots applied to such a state can only affect that
   result for future epochs, so are viable for querying target epoch.
   `compute_activation_exit_epoch(state.slot.epoch) > target.epoch`

2. Determine highest common ancestor among `state` and `target.blck`.
   At the ancestor slot, same rules re `get_active_validator_indices`.
   `compute_activation_exit_epoch(ancestorSlot.epoch) > target.epoch`

3. We now have a `state` that shares history with `target.blck` up
   through a common ancestor slot. Any blocks / slots that the `state`
   contains, which are not part of the `target.blck` history, affect
   `get_active_validator_indices` at epochs _after_ `target.epoch`.

4. Select `state.randao_mixes[N]` that is closest to common ancestor.
   Either direction is fine (above / below ancestor).

5. From that RANDAO mix, mix in / out all RANDAO reveals from blocks
   in-between. This is just an XOR operation, so fully reversible.
   `mix = mix xor SHA256(blck.message.body.randao_reveal)`

6. Compute the attester dependent slot from `target.epoch`.
   `if epoch >= 2: (target.epoch - 1).start_slot - 1 else: GENESIS_SLOT`

7. Trace back from `target.blck` to the attester dependent slot.
   We now have the destination for which we want to obtain RANDAO.

8. Mix in all RANDAO reveals from blocks up through the `dependentBlck`.
   Same method, no special handling necessary for epoch transitions.

9. Combine `get_active_validator_indices` from `state` at `target.epoch`
   with the recovered RANDAO value at `dependentBlck` to obtain the
   requested shuffling, and construct the `ShufflingRef` without replay.

* more tests and simplify logic

* test with different number of deposits per branch

* Update beacon_chain/consensus_object_pools/blockchain_dag.nim

Co-authored-by: Jacek Sieka <jacek@status.im>

* `commonAncestor` tests

* lint

---------

Co-authored-by: Jacek Sieka <jacek@status.im>
2023-05-12 19:36:59 +02:00
Etan Kissling d263f7f0cb
fix SSZ response for `produceBlindedBlock` (#4943)
* fix SSZ response for `produceBlindedBlock`

In `produceBlindedBlock`, we sent the `ForkedBlindedBeaconBlock` when
requested to reply in SSZ format. However, expected result is just the
inner `ForkyBlindedBeaconBlock` together with `eth-consensus-version`.

Note: We do not use SSZ format in our VC for this endpoint at this time,
which explains why we haven't noticed earlier.

* fix Altair/Phase0
2023-05-12 15:40:45 +00:00
Jacek Sieka 51418a7894
Incremental pruning (#4887)
* Incremental pruning

When turning on pruning the first time the current pruning algorithm
will prune the full database at startup. This delays restart
unnecessarily, since all of the pruned space is not needed at once.

This PR introduces incremental pruning such that we will never prune
more than 32 blocks or the sync speed, whichever is higher.

This mode is expected to become default in a follow-up release.
2023-05-12 13:37:15 +03:00
Jacek Sieka 938d21f1ed
fix linking / compile warnings on pie/etc (#4939)
* fix linking / compile warnings on pie/etc

* oops

* too much nim of late

* quotes
2023-05-12 07:57:49 +00:00
Jacek Sieka 09a69c7b07
better batch sig verification failure message 2023-05-12 08:18:13 +02:00
tersec cc341e0ab7
remove unused definitions (#4937) 2023-05-11 14:39:36 +00:00
Jacek Sieka b3c6320d56
embed genesis states using incbin (#4905) 2023-05-11 11:11:00 +00:00
tersec 2fcc01f516
modify newPayload failure logging (#4930) 2023-05-11 13:58:25 +03:00
Etan Kissling 18bc47fe83
add Capella constants to REST config endpoint (#4925)
Capella constants were missing; if we want to support BLS changes from
VC, we need the new domain constant to check compatibility.
2023-05-11 12:53:02 +02:00
Etan Kissling e0f024c0f3
capella preset for Gnosis (#4936)
```
MAX_WITHDRAWALS_PER_PAYLOAD* = 8
MAX_VALIDATORS_PER_WITHDRAWALS_SWEEP* = 8192
```

https://github.com/gnosischain/configs/pull/11
2023-05-11 09:56:32 +00:00
Etan Kissling 22c298c07b
final portion of trivial v1.3.0 bumps (#4926)
* final portion of trivial v1.3.0 bumps

Updates unchanged logic to latest v1.3.0 consensus-specs refs.

```
https://github.com/ethereum/consensus-specs/(blob|tree)/(?!v1\.3\.0/)
```

* lint
2023-05-11 09:54:29 +00:00
henridf 5dfd814588
Load trusted setup (#4870)
* Kzg: Load trusted setup

* scripts/launch_local_testnet.sh: set FIELD_ELEMENTS_PER_BLOB

* Use right setup file for mainnet/minimal

* Force rebuild

* Add comment explaining why build with -f
2023-05-11 11:52:44 +03:00
Etan Kissling e44b51e955
fix `attachMerkleProofs` to support multiple deposits (#4932)
`attachMerkleProofs` is used by `mockUpdateStateForNewDeposit` to create
a single deposit. The function doesn't work correctly when trying with
with multiple deposits, though. Fix this to enable more complex tests,
and also return the `deposit_root` for forming matching `Eth1Data`.
2023-05-11 11:45:55 +03:00
Etan Kissling a09b05bc27
fix `findShufflingRef` slot compute around genesis (#4910)
`dependent_slot` for epoch 2 is epoch 1's start slot - 1, not 0.
2023-05-11 11:39:40 +03:00
tersec a35c2c15b0
remove unused getBlock function (#4931) 2023-05-11 11:23:09 +03:00
henridf be3f5b1eac
More blob tweaks/fixes from running in devnet (#4933)
* BeaconNode: don't call fetchMissingblobs with empty list

* More logging

* BlockProcessor.checkBloblessSignature: Add missing return value
2023-05-11 00:36:35 +00:00
Etan Kissling 322429b191
final portion of non-trivial v1.3.0 bumps (#4927)
* final portion of non-trivial v1.3.0 bumps

Updates unchanged logic to latest v1.3.0 consensus-specs refs,
and cleans up surrounding sections / syncs comments, and so on.

```
https://github.com/ethereum/consensus-specs/(blob|tree)/(?!v1\.3\.0/)
```

* lint

* linebreak
2023-05-10 16:04:48 +02:00
tersec d3929cbb45
update some beacon API spec URLs; fix some Name and DuplicateModuleImport hints (#4929) 2023-05-10 10:20:55 +00:00
Etan Kissling 4873f8bdc4
update weak subjectivity calculations (#4923)
Weak subjectivity logic (`--weak-subjectivity-checkpoint`) was outdated.
Updated to latest specs, also taking into account total active balance.

See https://github.com/ethereum/consensus-specs/pull/2190
2023-05-10 08:23:59 +00:00
tersec e503cb4f51
make invalid execution payloads more visible (#4906) 2023-05-10 07:17:15 +00:00
Etan Kissling f4c215c5e6
cleanup `state_transition_epoch` and bump to v1.3.0 (#4922)
* cleanup `state_transition_epoch` and bump to v1.3.0

More v1.3.0 consensus-specs bumps, focused on `state_transition_epoch`.
Also fixed `current_epoch` spurious style check warning, and cleanup.

* Update beacon_chain/spec/state_transition_epoch.nim
2023-05-10 04:31:23 +02:00
Etan Kissling 34e7181b0d
`KZG(Commitment(s)?|Proof)` > `Kzg$1` (#4917)
Fix style-check by using matching capitalization for `nim-kzg` types.
2023-05-09 19:51:36 +00:00
Etan Kissling 93899b2e2a
consensus-specs v1.3.0 bumps with cleanup (#4918)
Some more consensus-specs v1.3.0 bumps. No semantic logic changes,
but also contains cleanup, so separate PR to reduce noise.
2023-05-09 20:16:49 +02:00
Etan Kissling e9319a5947
`bls_to_execution_changes` check alternative (#4916)
Make intent clearer about when to expect `bls_to_execution_changes`,
and replace test-time errors with compile-time errors if the field
is renamed or goes away in the future.
2023-05-09 17:24:02 +00:00
Etan Kissling b04970189b
more v1.3.0 consensus-specs bumps (#4915)
More spec references for unchanged logic bumped to v1.3.0.
2023-05-09 17:20:37 +00:00
Etan Kissling 5c5fbc088f
simplify FC weight computation (#4892)
Optimizations from https://github.com/ethereum/consensus-specs/pull/3246
avoid a couple unnecessary divisions when calculating proposer boost.
2023-05-09 10:16:13 +00:00
Etan Kissling 5b3c211285
`checkedReject` > `errReject` for CI failure (#4909)
The `SignedContributionAndProof: invalid contribution signature` check
is sometimes hit around fork boundaries when running local testnet.
To avoid failing CI, revert this isntance to a plain `errReject` until
the underlying problem is addressed.
2023-05-09 09:44:24 +00:00
zah 40253a76dd
Version 23.5.0 (#4913) 2023-05-09 11:17:18 +03:00
zah 5bf9284e62
Initial public version of the Verifying Web3Signer functionality (#4912)
* Allow the list of proved properties for web3signer to be configured
* Document the Web3Signer setups (regular, distributed and verified)
2023-05-09 11:16:43 +03:00
tersec 8f9bb391a3
don't consider attempt to route duplicate block an error (#4904) 2023-05-08 14:59:13 +02:00
henridf 53436c2b9b
Add blob validation condition (#4902) 2023-05-06 20:09:17 +00:00
Etan Kissling a23252c297
reorder `get_initial_beacon_block` by fork (#4899)
Deneb was listed between Capella and Bellatrix. Swap with Capella.
2023-05-06 10:32:59 +00:00
henridf 23adf15e5a
Blob handling sync fixes (#4888)
* Fix groupBlobs

* Fix getShortMap

* Fix blob handling in sync

* lint

* Add some blob-related logging
2023-05-06 08:58:50 +00:00
zah ae46be7020
Sign the blinded blocks only if they provide better value than the EL block (#4894) 2023-05-06 10:32:30 +02:00
Etan Kissling 1ebcd8b473
`excess_data_gas` after `withdrawals` cleanup (#4900)
We already updated the field order in the actual `ExecutionPayload`,
but in init code and tests / logs etc we still used the old order.
Update those occurrences to also match the field order in the struct.
Furthermore, add `excess_data_gas` to last entry in `test_eth1_monitor`.
2023-05-05 23:15:47 +00:00
Etan Kissling 297881edb7
bump gossip validation refs to 1.3.0 spec (#4895)
Updates gossip validation spec references to v1.3.0 and fixes an
incorrect reference to "signed_aggregate_and_proof" in sync contribution
documentation.
2023-05-05 22:48:33 +02:00
Etan Kissling 508819633a
more consensus-specs v1.3.0 bumps (#4898)
Updates more consensus-specs references to v1.3.0.
This PR only includes updates for sections otherwise unmodified.
2023-05-05 20:38:51 +00:00
Etan Kissling e408c7573d
bump more consensus-specs refs to v1.3.0 (with comments) (#4901)
Bump some more consensus-specs links to v1.3.0, and also align comments
with latest specs where they differ, and fix typos.
2023-05-05 20:37:56 +00:00
Etan Kissling 28968c11ba
remove `SAFE_SLOTS_TO_UPDATE_JUSTIFIED` (#4897)
The `SAFE_SLOTS_TO_UPDATE_JUSTIFIED` constant is no longer used as the
bouncing attack fix was removed:
https://github.com/ethereum/consensus-specs/pull/3290

Note: Some test networks still define the constant, ignoring the config
constant for now until it is no longer used.
2023-05-05 18:11:38 +00:00
Etan Kissling 0ad9ab446d
cleanup FC spec refs and align closer (#4893)
Bump FC spec references to v1.3.0, and visually align closer with
specs where trivial to do so without structural changes.
2023-05-05 17:03:54 +00:00
Zahary Karadjov d784672c10
Always use fresh HTTP connections for EL communication 2023-05-05 02:27:26 +03:00
tersec 2bd9e5a54c
log/read error from correct failed future (#4890) 2023-05-04 19:53:25 +03:00
zah 1e7eec18ce
Bugfix: Invalid execution header HTR leads to invalid state root (#4889) 2023-05-04 19:52:43 +03:00
Etan Kissling 4a1b94bc2e
update Deneb ANSI art for readability 🐟 (#4885)
Make "L" characters wider to improve readability.

Thanks to http://beatscribe.com (beatscribe#1008 on Discord)
2023-05-03 22:25:01 +02:00
Etan Kissling b5b51de67c
align `BeaconState` docs with latest spec (#4884)
Only comment changes:

- Bump refs to final v1.3.0 spec
- Align documentation style in various `BeaconState` structures
- Add `justification_bits` / `historical_roots` comment from spec
- Remove `previous_justified_checkpoint` from non-phase0 (same as spec)
- Cleanup some `Modified` tags
2023-05-03 13:05:46 +02:00
Zahary Karadjov 18d1f62ff1
Add logging for EL request time outs 2023-05-02 16:52:26 +03:00
Eugene Kabanov 0c75f9c2df
VC: Fix issue when optimistically synced node could block sync committee service functions. (#4878)
* Fix issue when optimistically synced node could block sync committee service functions.

* Update copyright year.
2023-05-02 14:54:44 +03:00
Etan Kissling ecaf6c8e92
harden gossip `strictVerification` in local testnets (#4880)
Fail local testnets on any gossip REJECT, instead of just asserting some
of the attestation related checks. This now also ensures that blocks,
BLS to Execution changes, blob sidecars and LC messages are checked
when running in a local testnet environment (`--verify-finalization`).

https://github.com/status-im/nimbus-eth2/pull/2904#discussion_r719603935
2023-05-02 13:06:02 +02:00
tersec dc32add555
only trigger liveness failsafe for mainnet (#4882)
* only trigger liveness failsafe for mainnet

* more user-friendly log
2023-05-01 11:45:39 +02:00
Etan Kissling bb62fee7d5
fix gossip validation for duplicate blob sidecars (#4881)
`SignedBlobSidecar: already have blob with valid signature` is IGNORE
in spec, but was implemented as REJECT. Align with spec.
2023-05-01 08:08:39 +02:00
henridf 5ccb085b9b
block_quarantine.nim: remove unused import (#4877)
leftover from #4844
2023-04-28 22:41:55 +00:00
henridf a32054cdd1
BlobQuarantine.hasBlobs: fix for loop bound (#4876) 2023-04-28 19:27:28 +00:00
henridf cf202fb928
Update nim-web3 (#4875)
Update to new version of nim-web3 incorporating the execution API
change that merges getPayloadV3 and getBlobsBundleV1.
2023-04-28 19:20:25 +00:00
henridf ef0b95dfbc
RequestManager: add support for fetching Blobs (#4844)
* RequestManager: add support for fetching Blobs

* Review feedback

* Lint

* Change peekSortedBlobless -> peekBlobless
2023-04-28 12:57:35 +00:00
Zahary Karadjov 3b58711e6b
Merge branch 'stable' into unstable 2023-04-28 00:12:37 +03:00
Zahary Karadjov e6a6019ef9
Fix the cross-compilation for Windows 2023-04-27 16:28:10 +03:00
Etan Kissling dd1ffa5ded
add `el_offline` to `/eth/v1/node/syncing` (#4860)
Add compatibility with https://github.com/ethereum/beacon-APIs/pull/290
to the beacon node. Behaviour when configured with multiple ELs is not
specified; intention suggests to indicate whether all ELs are offline.
2023-04-27 10:47:38 +02:00
henridf ea8ef5799e
Fill in missing BlobSidecar gossip check (#4863) 2023-04-26 17:33:33 +00:00
Etan Kissling 107f733365
make `--trusted-block-root` option visible for CP sync (#4859)
Add `--trusted-block-root` to `--help` for `trustedNodeSync`.
2023-04-26 13:37:58 +02:00
Etan Kissling c4e258af2a
check signatures in CP sync backfill with LC (#4858)
When using trusted node sync with light client (`--trusted-block-root`),
the trust assumption on the server is reduced to solely be responsible
for data availability, but not data correctness. This means that we must
check block proposer signatures against the downloaded checkpoint, as
they are not covered by the block root.

Note that this lowers the backfill speed when using LC based CP sync
due to the extra checks, by about 60% for me.
2023-04-26 13:37:27 +02:00
tersec 1ccb36b272
include small dedup in block processor to handle blockByRoot blocks (#4850) 2023-04-26 07:00:03 +00:00
zah 55ae7d3b00
Version 23.4.0 (#4857)
* Draft release notes for version 23.4.0

* Drop the reference to the Builder API failsafe spec PR
2023-04-25 22:46:38 +03:00
henridf da6169bfe9
BlockProcessor.storeBlock: write blobs to DB (#4854) 2023-04-25 13:55:35 +03:00
zah 9b2c07c118
Allow exiting multiple validators at once (#4855) 2023-04-25 09:44:01 +03:00
Jacek Sieka 58b93ccbe0
era: Capella+ support (fixes #4752) (#4853)
Post-Capella, historical roots are computed from historical summaries
instead of being directly stored in the beacon state.

Slightly messy to pass both lists around - this is done to avoid
computing the historical root unnecessarily.
2023-04-24 15:26:28 +02:00
tersec fbe90dcbea
update statediffs to support capella (#4852) 2023-04-23 22:15:14 +03:00
tersec ed7ad56d16
allow for biasing comparisons between engine/builder api bids (#4847) 2023-04-23 22:10:34 +03:00
tersec b390911e93
automatic update of v1.3.0-rc.5 consensus spec URLs to v1.3.0 (#4848) 2023-04-21 18:52:43 +00:00
tersec c0b90833db
don't crash on block production error; build lookahead proposal from current head (#4842) 2023-04-20 15:07:27 +00:00
tersec d3400ca11b
low attestations during epoch should instafail in CI; dbg -> warn level in `newPayload` log (#4830)
* low attestations during epoch should instafail in CI; dbg -> warn level on newPayload log

* improve newPayload warning message when no valid EL connected

* reduce potential spam; make log spelling more consistent; use fatal/quit
2023-04-19 19:42:30 +00:00
tersec 2246a6ec95
Revert "include small dedup in block processor to handle blockByRoot blocks (#4814)" (#4840)
This reverts commit 8b3ffec0d5.

Syncing was broken with this: https://github.com/status-im/infra-nimbus/issues/132#issuecomment-1514465481
2023-04-19 19:16:27 +00:00
henridf 12d640b691
Request manager: Handle blobless blocks (#4833)
Post-Deneb, when the request manager receives a missing block from a
peer, it needs to check if the corresponding blobs are available, and
if so pass them along. If they aren't available, the newly-fetched
block must be put in blobless quarantine (while the blobs are
retrieved, coming in next commit).
2023-04-19 19:37:38 +03:00
tersec 0fc89484de
ignore EL exchangeTransitionConfiguration results (#4831) 2023-04-19 12:16:50 +00:00
tersec 4643a357ef
consensus spec ref URL update to v1.3.0-rc.5 (#4837) 2023-04-18 22:00:06 +00:00
tersec 45f89f1205
use consensus specs v1.3.0 test vectors (#4838) 2023-04-18 22:01:49 +02:00
Etan Kissling e6e4ba9de6
clean up redundant tests and config (#4836)
The consensus-spec-tests already cover the scenarios of our custom test
runner, so the custom tests can be removed. Also cleans up unused config
flags and related unreachable logic.
2023-04-18 21:26:36 +02:00
tersec 77c0aa713e
use non-nil head corresponding to execution head (#4835) 2023-04-18 14:17:56 +00:00
Etan Kissling 1d3e838236
lower log level of LC optimistic sync (#4834)
When using light client sync while main forward sync is behind, use
DEBUG log level instead of INFO log level to reduce log verbosity.
2023-04-18 13:37:29 +00:00
tersec cc24429828
remove empty block fallback now that capella's on mainnnet (#4821)
* remove empty block fallback now that capella's on mainnnet

* build_empty_execution_payload is only testing infrastructure now

* update error message
2023-04-18 09:21:15 +00:00
henridf 8f2f40f0af
Remove blockBlobsVerifier (#4829)
Having a separate 'blockBlobsVerifier' function for post-Deneb blocks
is no longer of any benefit. This commit removes it.
2023-04-18 02:12:57 +02:00
Eugene Kabanov b51152153a
VC: Hardening and optimizing time handling. (#4743)
* Fix durationToNextSlot() and durationToNextEpoch() to work not only after Genesis, but also before Genesis.
Change VC pre-genesis behavior, add runPreGenesisWaitingLoop() and runGenesisWaitingLoop().
Add checkedWaitForSlot() and checkedWaitForNextSlot() to strictly check current time and print warnings.
Fix VC main loop to use checkedWaitForNextSlot().
Fix attestation_service to run attestations processing only until the end of the duty slot.
Change attestation_service main loop to use checkedWaitForNextSlot().
Change block_service to properly cancel all the pending proposer tasks.
Use checkedWaitForSlot to wait for block proposal.
Fix block_service waitForBlockPublished() to be compatible with BN.
Fix sync_committee_service to avoid asyncSpawn.
Fix sync_committee_service to run only until the end of the duty slot.
Fix sync_committee_service to use checkedWaitForNextSlot().

* Refactor validator logging.
Fix aggregated attestation publishing missing delay.

* Fix doppelganger detection should not start at pre-genesis time.
Fix fallback service sync status spam.
Fix false `sync committee subnets subscription error`.

* Address review comments part 1.

* Address review comments.

* Fix condition issue for near genesis waiting loop.

* Address review comments.

* Address review comments 2.
2023-04-17 21:31:54 +00:00
tersec 228e10f1d9
update engine API URLs from v1.0.0-beta.2 to beta.3 (#4828) 2023-04-17 20:11:28 +00:00
tersec 8b3ffec0d5
include small dedup in block processor to handle blockByRoot blocks (#4814) 2023-04-17 19:36:15 +00:00
Jacek Sieka 7df75d77fa
move spec constants to spec (#4825)
* move spec constants to spec

quite funny, this removes >_1000_ transitive imports when building
`nimbus_validator_client`.

`genDepend`:

```sh
[arnetheduck@praeceps beacon_chain]$ diff -u
nimbus_validator_client.dot.pre nimbus_validator_client.dot | wc
   1167    3507  103822
```

* imports

* better constants

* oops

* too many blobs

* reorder spec constants in fork order

* cleanup
2023-04-17 17:18:54 +00:00
henridf 176c80a3d5
Fix incorrect naming and use of Quarantine.peekBlobless (#4826)
The 'peek' name was incorrect as it was actually removing from the
table. It was consequently used incorrectly in block processing: the
blobless block wasn't returned to the table when it should be.
2023-04-17 18:10:13 +02:00
tersec 75be7d267d
always use fcUV2 in shapella even for non-proposer fcUs (#4817)
* always use fcUV2 in shapella even for non-proposer fcUs

* avoid template/proc naming conflict with libp2p/signed_envelope.nim having a payload proc
2023-04-17 16:17:52 +02:00
henridf 4df851f4f5
Fix incorrect handling of blobless blocks (#4822)
We were passing along all blocks to local processing, including those
for which we were missing blobs. This commit fixes that.
2023-04-17 10:54:04 +00:00
henridf 29b431e312
Simplify block quarantine blobless (#4824)
* Simplify block quarantine blobless

The quarantine blobless table was initially keyed off of (Eth2Digest,
ValidatorSig). This was modelled off the orphan table. The presence of
the signature in the key is necessary for orphans, because we can't
verify the signature for an orphan. That is not the case for a
blobless block, where the signature can be verified.

So this PR changes the blobless block table to be keyed off a
Eth2Digest only. This simplifies the retrieval and handling of
blobless blocks.

* review feedback
2023-04-16 08:37:56 +00:00
Etan Kissling cb9e0eed49
allow trusted node sync based on LC trusted block root (#4736)
* allow trusted node sync based on LC trusted block root

Extends `trustedNodeSync` with a new `--trusted-block-root` option that
allows initializing a light client. No `--state-id` must be provided.
The beacon node will then use this light client to obtain the latest
finalized state from the remote server in a trust-minimized fashion.
Note that the provided `--trusted-block-root` should be somewhat recent,
and that security precautions such as comparing the state root against
block explorers is still recommended.

* fix

* workaround for `valueOr` limitations

* reduce magic numbers

* digest len > context len for readability

* move `cstring` conversion to caller

* avoid abbreviations

* `return` codestyle
2023-04-16 06:07:07 +00:00
henridf 57623af36a
Remove unnecessary field derefs in BlockProcessor.storeBlock (#4823) 2023-04-16 01:25:17 +00:00
tersec f6062a2be1
compute correct withdrawals for lookahead proposal fcUs at epoch boundaries (#4820) 2023-04-15 21:48:24 +00:00
tersec ae45e11f8e
log when attempting to validate with exited validators (#4812) 2023-04-15 21:31:24 +00:00
henridf 021de18e06
Quarantine and reassembly of gossiped blobs and blocks (#4808) 2023-04-13 19:11:40 +00:00
Etan Kissling fa3655527c
add `formatIt` for `ForkedLightClientXyz` (#4816)
* add `formatIt` for `ForkedLightClientXyz`

Even though the `ForkyLightClientXyz` have `formatIt`, they do not apply
when logging `ForkedLightClientXyz`, leading to large logs at times.
Defining `formatIt` for `ForkedLightClientXyz` fixes this.

* exports not needed
2023-04-13 15:26:17 +00:00
Miran 0ec7df1042
fix spelling in conf.nim (#4813) 2023-04-12 17:01:29 +00:00
tersec cd7da00d16
eliminate fcU/getPayload race condition causing missed proposals (#4800) 2023-04-12 12:33:21 +03:00
Etan Kissling 56986c08d6
rename `execution(Block|Payload)Root` > `executionBlockHash` (#4809)
Note: `execution_payload_root` is _actually_ `htr(payload)`.
Only `executionPayloadRoot` was used as `executionBlockHash`.
2023-04-11 23:31:47 +00:00
Etan Kissling dc22244b3e
fix `/eth/v1/debug/fork_choice` output (#4810)
Two fixes to `/eth/v1/debug/fork_choice`:

- `validity` enum is expected to be serialized as string instead of int
- `data` wrapper is not expected for this endpoint
2023-04-11 23:17:48 +00:00
Etan Kissling c3d043c0e1
rename `loadExecutionBlockRoot` > `loadExecutionBlockHash` (#4807)
There are still some `executionBlockRoot` after this, separate rename.
2023-04-11 16:56:29 +00:00
tersec eed34e740a
implement the getStateRandao Beacon API (#4799) 2023-04-11 18:27:48 +03:00
tersec 0fbf911722
use higher of available engine and builder API bids (#4795) 2023-04-11 18:19:48 +03:00
henridf a543b0b446
Quarantine.cleanupOrphans(): Delete unviable orphan (#4782) 2023-04-11 08:51:01 +02:00
Etan Kissling df7ecd4fe9
update `/eth/v1/debug/fork_choice` format to latest (#4802)
Syncs the `/eth/v1/debug/fork_choice` REST endpoint with latest specs.

- Validity is now reported as tri-state `enum` instead of two `bool`s
- Response includes store's justified and finalized checkpoints
- Additional `ExtraData` field on outer layer (empty for now)

https://github.com/ethereum/beacon-APIs/pull/232
2023-04-10 15:35:14 +00:00
Etan Kissling 450f06566b
accelerate execution layer sync using light client (#4805)
Turn on `--sync-light-client` option by default, now that it has shown
stability in local testnets.
2023-04-10 14:28:46 +00:00
Etan Kissling 566c178e59
add ANSI art for Deneb 🐟 (#4803)
Trigger ANSI art on upgrade to Deneb

Art created by http://beatscribe.com (beatscribe#1008 on Discord)
2023-04-10 12:42:19 +00:00
Etan Kissling b7d08d0a38
do not report pre-Merge sync progress as `/opt` (#4801)
Before the merge, assume `payloadStatus == NewPaylodStatus.valid` to
avoid cases of sync progress being reported with `/opt` suffix.
2023-04-09 14:58:20 +00:00
tersec 7857c47a8f
fix expected withdrawal mismatch logging (#4798) 2023-04-08 09:15:09 +00:00
tersec 742de1347d
only add active validators to knownValidators at startup (#4793) 2023-04-06 13:53:06 +00:00
Eugene Kabanov 0ff86e9538
web3signer refactoring and test suite. (#4775)
* Refactor nimbus_signing_node to support Unix signals.

* Fix SN unable to close REST server properly.

* Fix `keys`, `deposit` and `validator_registration` endpoints issues.
Add getValidatorExitSignature() and getDepositMessageSignature() to validator_pool.

* Add /reload endpoint and implementation.
Fix signData to not cancel `timer`.
Fix validator_pool should clear attachedValidators table.

* Diva protocol enhancement implementation.
2023-04-06 16:16:21 +03:00
Zahary Karadjov d5f454a8f5
Fix a logical typo 2023-04-06 12:53:44 +03:00
zah b59f9f5e1a
Remove some doAsserts that are easy to trigger from user actions (#4791) 2023-04-05 18:52:42 +00:00
tersec 3bfff6f219
query EL and builder relay for bids in parallel (#4749)
* outline for comparing bids from builder and engine API in BN

* set up async

* decision scaffold

* clean up logging

* Refactor proposeBlockMEV

* Update beacon_chain/validators/validator_duties.nim

Co-authored-by: zah <zahary@status.im>

* Update beacon_chain/validators/validator_duties.nim

Co-authored-by: zah <zahary@status.im>

* use typedescs instead of explicit generic parameters

---------

Co-authored-by: zah <zahary@status.im>
2023-04-05 16:35:32 +03:00
henridf 04302081b4
Add blob sidecar gossip validation (#4785)
* Add blob gossip validation

* lint

* Add test for getBlobSidecarTopic

* Fix closure variable capture issue

* Update beacon_chain/nimbus_beacon_node.nim

Co-authored-by: tersec <tersec@users.noreply.github.com>

---------

Co-authored-by: tersec <tersec@users.noreply.github.com>
2023-04-04 15:12:34 +00:00
Tanguy 5167a373ab
Bump libp2p (#4786)
* auto-bump nim-libp2p

* remove iWantBudget

---------

Co-authored-by: = <Menduist@users.noreply.github.com>
2023-04-04 13:31:41 +00:00
Eugene Kabanov e2bf58a6f2
Fix issue when VC crashes on invalid `beacon-node` argument. (#4765)
Make `beacon-node` URL argument less strict and more UX friendly.
2023-04-03 16:25:16 +00:00
tersec 464e680efc
detect missing network definitions at compile-time (#4784) 2023-04-03 15:39:12 +00:00
Eugene Kabanov 9e5c19eba5
Adjust remote signer idle connection timeout defaults. (#4779)
* Adjust remote signer idle connection timeout defaults.

* Address review comments.
2023-04-02 10:38:28 +00:00
tersec c31fbc3977
remove most std/options imports (#4778) 2023-03-31 20:46:47 +00:00
tersec b4508b853a
assorted consensus spec URL updates (#4776) 2023-03-30 23:08:52 +00:00
Zahary Karadjov 499b870a1a
Allow the git revision to be specified with -d:git_revision_override=xxxxxx 2023-03-30 20:01:20 +03:00
tersec f71a279d17
more deneb support in ncli_db and forks (#4774) 2023-03-30 10:06:23 +00:00
tersec 0d051b3da8
add Deneb support to REST (de)serialization (#4770) 2023-03-29 11:41:54 +03:00
tersec aa04d45335
use canonical, standalone sepolia network definition repository (#4762) 2023-03-29 11:37:14 +03:00
tersec 9d2fdc2e26
assorted consensus spec URL updates to v1.3.0-rc.5 (#4768) 2023-03-27 14:41:22 +00:00
tersec 29c77948cb
document rationale for await/awaitWithTimeout selection (#4764) 2023-03-23 19:54:41 +00:00
tersec 69bc9bcd85
no timeout for purely local async slashing protection and signing (#4751) 2023-03-23 16:51:55 +02:00
henridf 635a924e8c
Add nim-kzg4844 and use it in validate_blobs (#4732) 2023-03-23 09:47:04 +00:00
tersec fc1f9a2065
builder API liveness failsafe (#4746)
* builder API liveness failsafe

* add test summary change
2023-03-22 18:48:48 +01:00
Zahary Karadjov 6c0d756d54
Version 23.3.2 2023-03-22 01:50:47 +02:00
tersec 44b38ef137
Implement /eth/v1/validator/{beacon,sync}_committee_selections Beacon API end-points (#4760) 2023-03-22 01:47:36 +02:00
zah eb678be0ff
Don't report an 'EL connection degraded' warning when the EL is not synced (#4761) 2023-03-22 00:19:36 +02:00
zah b8f2022668
Remove an overly agressive assertion (#4759) 2023-03-21 23:35:22 +02:00
Zahary Karadjov 813fddee8c
Capella hard-fork epoch for Mainnet 2023-03-21 23:33:06 +02:00
Zahary Karadjov 865c84cc18
Some documentation for the new 'deposit exit' work-around 2023-03-21 23:06:57 +02:00
cheatfate 596006be08
Workaround for issue #4216. 2023-03-21 23:06:57 +02:00
Tanguy 769ed00203
Add light client gossip metrics (#4745) 2023-03-21 08:55:48 +01:00
tersec 2f634c10a4
automated consensus spec URL updating from v1.3.0-rc.4 to rc.5 (#4756) 2023-03-21 00:42:22 +00:00
tersec 82bf949561
use consensus spec v1.3.0-rc.5 test vectors (#4754) 2023-03-20 20:17:25 +00:00
Zahary Karadjov 4d1b2dd9f5
Merge branch 'stable' into unstable 2023-03-17 17:51:39 +02:00
tersec 0c62950de6
more consensus spec v1.3.0-rc.4 URL updates (#4744) 2023-03-17 12:14:08 +00:00
tersec ec77116414
automated consensus spec URL updating from v1.3.0-rc.3 to rc.4 (#4742) 2023-03-17 01:10:31 +00:00
Etan Kissling fb26e19de9
use v1.3.0-rc.4 consensus spec test vectors (#4740) 2023-03-16 21:35:20 +01:00
Zahary Karadjov 3615606914
Version 23.3.1 2023-03-14 19:43:11 +02:00
Zahary Karadjov ee610cbf34
Restore the hidden --web3-force-polling option as a deprecated one 2023-03-14 18:54:15 +02:00
Zahary Karadjov 957a8c2fdc
Bugfix: A single unsynced EL client may cause an inappropriate switch to optimistic mode. 2023-03-14 18:19:11 +02:00
Zahary Karadjov d899a6a834
Prevent a potential panic on dereferencing a 'none' value 2023-03-14 18:05:29 +02:00
Zahary Karadjov 46f48269ef
Backwards compatible handling of the web3-url parameter in TOML 2023-03-14 17:50:03 +02:00
Zahary Karadjov 3a35809a02
Backwards-compatible handling of Engine URLs that don't include a specified protocol 2023-03-14 14:02:50 +02:00
henridf 9906a0d5a2
Gate Deneb `beacon_block` subscription (#4735) 2023-03-14 11:52:01 +00:00
Zahary Karadjov 2ee15a38ed
Reduce the number of block timestamp requests
The timestamps are only relevant within the candidate block range.
2023-03-14 13:47:13 +02:00
Zahary Karadjov 6fb48aca7d
Lower the maximum block range in getLogs requests to 1000 2023-03-14 13:45:01 +02:00
Etan Kissling ceefc49965
rm redundant `ConsensusFork.toString` (#4733) 2023-03-14 01:24:10 +00:00
Etan Kissling 5ef1ed5d96
indicate Capella / Shanghai readiness on startup (#4630)
In startup "Launching beacon node" message, log readiness for Capella /
Shanghai as part of config to aid with support requests.
2023-03-13 22:01:15 +01:00
Etan Kissling 29a699f69d
rename `blockFork` > `consensusFork` (#4731)
* rename `blockFork` > `consensusFork`

* revert one overzealous stateFork / blockFork rename
2023-03-12 18:48:38 +00:00
Etan Kissling f0dcbaacff
extend `addTestBlock` with Capella/Deneb support (#4728)
* extend `addTestBlock` with Capella/Deneb support

* bump copyright years

* fix
2023-03-11 20:11:33 +00:00
Etan Kissling fd8e86972d
rename `lcDataForkAtStateFork` > `lcDataForkAtConsensusFork` (#4726) 2023-03-11 20:09:21 +00:00
Etan Kissling 3e872bec86
rename `eip4844_preset.nim` > `deneb_preset.nim` (#4724) 2023-03-11 17:00:35 +00:00
Etan Kissling d8f39dd24f
rename `stateForkForDigest` > `consensusForkForDigest` (#4727) 2023-03-11 16:58:48 +00:00
Etan Kissling b0cccae7f4
rename `(with|at)StateFork` > `(with|at)ConsensusFork` (#4725) 2023-03-11 14:39:29 +00:00
Etan Kissling eba5f9af26
rename `upgrade_to_eip4844` > `upgrade_to_deneb` (#4723) 2023-03-11 14:28:55 +00:00
Etan Kissling e542d079b0
rename Deneb database tables (not in testnets atm) (#4720) 2023-03-11 02:26:03 +01:00
Etan Kissling 69013d153c
bump light client spec references to `v1.3.0-rc.3` (#4719) 2023-03-11 01:11:51 +00:00
Zahary Karadjov 17c0eeeede
Version 23.3.0 2023-03-11 02:51:44 +02:00
Etan Kissling ad118cd354
rename `stateFork` > `consensusFork` (#4718)
Just the variable, not yet `lcDataForkAtStateFork` / `atStateFork`.

- Shorten comment in `light_client.nim` to keep line width
- Do not rename `stateFork` mention in `runProposalForkchoiceUpdated`.
- Do not rename `stateFork` in `getStateField(dag.headState, fork)`

Rest is just a mechanical mass replace
2023-03-11 00:35:52 +00:00
Etan Kissling 969c6f73ae
misc local `EIP4844` > `Deneb` bumps (#4717)
* misc local `EIP4844` > `Deneb` bumps

* fix
2023-03-11 00:28:19 +00:00
Zahary Karadjov f91fe5646e
Redo fix #4588 2023-03-10 19:40:27 +02:00
Zahary Karadjov 89dccedcc9
Prevent the build up of parallel 'waitELToSync' operations 2023-03-10 19:24:20 +02:00
Etan Kissling 57b2151f95
rename `EIP4844` > `Deneb` for light client (#4713)
* rename `EIP4844` > `Deneb` for light client

* regenerate test logs
2023-03-10 17:16:26 +00:00
Etan Kissling 8fd3723e82
rename `EIP4844BeaconStateNoImmutableValidators` > `DenebXyz` (#4712) 2023-03-10 17:15:08 +00:00
Etan Kissling b0577bdc6c
rename `OnEIP4844BlockAdded` > `OnDenebBlockAdded` (#4714) 2023-03-10 17:14:27 +00:00
Etan Kissling e4b5478fff
rename `ForkDigests.eip4844` > `ForkDigests.deneb` (#4715) 2023-03-10 17:13:40 +00:00
henridf f5612f2a77
Remove BlobsSidecar used in BeaconChainDB (#4710) 2023-03-10 12:51:36 +00:00
Zahary Karadjov 70d3045584
Cosmetic logging changes in the EL manager 2023-03-10 11:12:29 +02:00
zah c6ce7f383d
More forceful EL connection resetting (fixes #4424) (#4708) 2023-03-09 23:41:28 +00:00
Zahary Karadjov f5a3ea6cbb
Fix an accidentally wrong partial git stage 2023-03-09 22:30:01 +02:00
Zahary Karadjov e808fda02c
engine_api_response_time renamed to engine_api_request_duration_seconds 2023-03-09 19:29:24 +02:00
zah ef20e831a8
More metrics for the EL monitor (#4707)
* `engine_api_response_time` provides a histogram for the Engine API
  response times for each unique pair ot URL and request type.

* All engine API requests are now tracked

Other changes:

The client will no longer exit on start-up if it fails to connect to
a properly configured EL node.
2023-03-09 19:29:04 +02:00
tersec 8625704dde
don't clear validator change subpools after creating each block (#4704)
* don't clear validator change subpools after creating each block

* remove now-spurious tests of removed behavior
2023-03-09 18:19:36 +02:00
tersec a47f0b054e
finish eip4844 to deneb module rename (#4705) 2023-03-09 01:34:17 +01:00
Etan Kissling 879f8ab5e8
check correct fork version in LC sync protocol (#4703)
Tests updated: https://github.com/ethereum/consensus-specs/pull/3284
2023-03-08 19:59:21 +01:00
Etan Kissling e499c709f4
avoid pruning LC data pre dag.tail.slot (#4702)
When using `--history=prune`, `dag.tail.slot` may advance beyond the
configured light client data retention period. Update the LC logic so
that the `dag.tail.slot` is no longer considered for LC pruning.
It is still considered to check whether new data can be produced.
2023-03-08 19:59:00 +01:00
henridf 90640cce05
Update sync to use post-decoupling RPC (#4701)
* Update sync to use post-decoupling RPCs

blob_sidecars_by_range returns a flat list of sidecars, which must
then be grouped per-slot.

* Add test for groupBlobs

* createBlobs: convert proc to func
2023-03-07 20:19:17 +00:00
tersec 2c0762013e
bump eth2-networks to include goerli/prater capella fork epoch (#4699) 2023-03-07 05:03:47 +00:00
zah 9cd30768e7
Bugfix: historic blocks violate the latest specs for the extraData field (fixes #4695) (#4700) 2023-03-07 04:43:27 +00:00
tersec 63b1b0840f
5 more modules of eip4844.foo to deneb.foo renames (#4698) 2023-03-06 18:45:52 +00:00
tersec 8541674498
simplify ELMonitor fcU payload attributes handling (#4696) 2023-03-06 16:19:15 +00:00
zah 8771e91d53
Support for driving multiple EL nodes from a single Nimbus BN (#4465)
* Support for driving multiple EL nodes from a single Nimbus BN

Full list of changes:

* Eth1Monitor has been renamed to ELManager to match its current
  responsibilities better.

* The ELManager is no longer optional in the code (it won't have
  a nil value under any circumstances).

* The support for subscribing for headers was removed as it only
  worked with WebSockets and contributed significant complexity
  while bringing only a very minor advantage.

* The `--web3-url` parameter has been deprecated in favor of a
  new `--el` parameter. The new parameter has a reasonable default
  value and supports specifying a different JWT for each connection.
  Each connection can also be configured with a different set of
  responsibilities (e.g. download deposits, validate blocks and/or
  produce blocks). On the command-line, these properties can be
  configured through URL properties stored in the #anchor part of
  the URL. In TOML files, they come with a very natural syntax
  (althrough the URL scheme is also supported).

* The previously scattered EL-related state and logic is now moved
  to `eth1_monitor.nim` (this module will be renamed to `el_manager.nim`
  in a follow-up commit). State is assigned properly either to the
  `ELManager` or the to individual `ELConnection` objects where
  appropriate.

  The ELManager executes all Engine API requests against all attached
  EL nodes, in parallel. It compares their results and if there is a
  disagreement regarding the validity of a certain payload, this is
  detected and the beacon node is protected from publishing a block
  with a potential execution layer consensus bug in it.

  The BN provides metrics per EL node for the number of successful or
  failed requests for each type Engine API requests. If an EL node
  goes offline and connectivity is resoted later, we report the
  problem and the remedy in edge-triggered fashion.

* More progress towards implementing Deneb block production in the VC
  and comparing the value of blocks produced by the EL and the builder
  API.

* Adds a Makefile target for the zhejiang testnet
2023-03-05 01:40:21 +00:00
tersec e3d96ef147
rename most eip4844Data to denebData (#4693) 2023-03-04 22:23:52 +00:00
tersec 3b41e6a0e7
rename ConsensusFork.EIP4844 to ConsensusFork.Deneb (#4692) 2023-03-04 13:35:39 +00:00
Eugene Kabanov c8b50765cf
Various fixes for VC and BN REST server. (#4673)
* Fix issue when VC unable to detect errors properly and act accordingly.
Switch all API functions used by VC to RestPlainResponse, this allows us to print errors returned by BN servers.

* Fix issue when prepareBeaconCommitteeSubnet() do not perform actions when BN is optimistically synced only.

* Fix Defect issue.

* Fix submit/publish returning `false` when operation was successful.

* Address review comments.

* Fix some client calls unable to receive `execution_optimistic` field, mark BN as OptSynced when such request has been made.

* Adjust warning levels.

---------

Co-authored-by: Jacek Sieka <jacek@status.im>
2023-03-03 20:20:01 +00:00
Jacek Sieka fec580c39c
doppelganger: guard against reordering (#4691) 2023-03-03 18:38:48 +01:00
Jacek Sieka 65aee9cf22
ncli_db: use fork from runtimeconfig when verifying era file (#4689)
Sepolia in particular goes through two hard forks during era 1, meaning
the phase0 fork is no longer part of the state, even though blocks that
belong to it still are phase0.
2023-03-03 13:15:11 +01:00
tersec ea060de6d4
more eip4844 -> deneb module references (#4690) 2023-03-02 21:09:24 +01:00
tersec d058aa09c8
more withdrowls (#4674) 2023-03-02 17:13:35 +01:00
Jacek Sieka 4d9eaafe9c
vc: fix missing attestations due to doppelganger in epoch 1 (#4688)
* log validator that triggers doppelganger
* move activity detection closer to where it's performed, record
aggregation as activity (since it's part of the liveness endpoint)
* vc: check for doppelgangers in epoch 0 also (so that activity in epoch
1 can happen)
* avoid some looping when processing activities
2023-03-02 16:55:45 +01:00
henridf 1de3cf5246
Remove SignedBeaconBlockAndBlobsSidecar (#4683)
This commit removes SignedBeaconBlockAndBlobsSidecar and all remaining
references.
2023-03-02 15:12:04 +01:00
tersec 88092bb411
don't try to validate execution block hashes of non-execution payloads (#4687) 2023-03-02 00:11:46 +00:00
henridf 5a9e63a029
Remove uses of beacon_block_and_blobs_sidecar topic (#4682)
* Remove use of beacon_block_and_blobs_sidecar topic

This topic goes away with decoupled blocks and blobs.

* remove use of getBeaconBlockAndBlobsSidecarTopic from test

* update nimbus_light_client.nim
2023-03-01 21:30:20 +01:00
henridf 3681177cf4
Remove ForkySignedBeaconBlockMaybeBlobs (#4681)
This commit removes ForkySignedBeaconBlockMaybeBlobs and all
references. I tried to pull that thread only as little as was needed
to get rid of it. Left a placeholder BlobSidecar array (in lieu of
Opt[BlobsSidecar]) in a few places; this will be used as we rebuild
the decoupled implementation.
2023-02-28 11:36:17 +00:00
tersec fb1b7458f0
almost all remaining automated v1.3.0-rc.2 to rc.3 consensus spec URL updates (#4679) 2023-02-27 21:37:37 +00:00
tersec 8ee5e8271d
weaken overaggressive different-fork-next-slot check (#4665) 2023-02-27 21:23:00 +01:00
henridf 0a87889eaf
Add blobSidecar RPCs (#4677)
* Add blob_sidecars_by_range RPC

* Add blob_sidecars_by_root RPC
2023-02-27 21:16:59 +01:00
henridf 144130e80e
Add DB support for BlobSidecar (#4675) 2023-02-27 14:02:37 +00:00
henridf dede36fe86
Remove blobsSidecar from orphans table (#4670) 2023-02-27 06:10:22 +00:00
henridf a999da3361
Remove beacon_block_and_blobs_sidecar_by_root RPC (#4672)
Along with its use in the request manager.
2023-02-26 23:17:40 +00:00
tersec 982d79f9a2
more eip4844 -> deneb changes (#4666) 2023-02-25 03:03:34 +02:00
tersec 2dd2a782d9
disable BareExcept warnings (#4662)
* disable BareExcept warnings

* BareExcept only exists in >= 1.6.11
2023-02-24 19:50:46 +00:00
Eugene Kabanov e21742a325
Fix attester and proposer duties optimistic execution behavior. (#4657)
* Fix attester and proposer duties optimistic execution behavior.

* Address review comments.

* Address review comments more efficiently.
2023-02-23 23:13:17 +00:00
tersec 069d973fc6
use 'deneb' in 4 more modules (#4658) 2023-02-23 19:06:57 +01:00
Eugene Kabanov dd911a7ac6
Fix IndexError for reasons. (#4656)
* Fix IndexError for reasons.

* Deprecate specific Timeout reason.
Log exception message instead of single Timeout error.
2023-02-23 17:02:17 +01:00
tersec 8f269c92d7
rename eip48844ImplementationMissing to denebImplementationMissing (#4654) 2023-02-23 10:37:45 +00:00
tersec c6cefd8914
add success logging for being configured to use payload builder (#4653) 2023-02-23 11:46:18 +02:00
zah 6036f2e7d7
Local sim impovements (#4551)
* Local sim impovements

* Added support for running Capella and EIP-4844 simulations
  by downloading the correct version of Geth.

* Added support for using Nimbus remote signer and Web3Signer.
  Use 2 out of 3 threshold signing configuration in the mainnet
  configuration and regular remote signing in the minimal one.

* The local testnet simulation can now use a payload builder.
  This is currently not activated in CI due to lack of automated
  procedures for installing third-party relays or builders.

  You are adviced to use mergemock for now, but for most realistic
  results, we can create a simple builder based on the nimbus-eth1
  codebase that will be able to propose transactions from the regular
  network mempool.

* Start the simulation from a merged state. This would allow us
  to start removing pre-merge functionality such as the gossip
  subsciption logic. The commit also removes the merge-forcing
  hack installed after the TTD removal.

* Consolidate all the tools used in the local simulation into a
  single `ncli_testnet` binary.
2023-02-23 02:10:07 +00:00
Eugene Kabanov 08b6bb7a6b
VC: Use not-synced/opt-synced BNs. (#4635)
* Initial commit.

* Address review comments and recommendations.

* Fix too often `Execution client not in sync` messages in logs.

* Add failure reason for duties requests.

* Add more reasons to every place of ValidatorApiError.

* Address race condition issue.

* Remove `vc` argument for getFailureReason().
2023-02-23 00:11:00 +00:00
tersec 1dd07d5def
switch 3 modules in beacon_chain/spec/ from eip4844 to deneb (#4651) 2023-02-22 14:10:00 +01:00
Jakub Sokołowski 1a0b2daa0a
beacon_chain: fix history flag help message (#4650)
Signed-off-by: Jakub Sokołowski <jakub@status.im>
2023-02-22 14:09:38 +01:00
tersec ffaa42b880
fix Deneb/EIP4844 phrasing (#4646) 2023-02-21 19:25:07 +01:00
tersec 29fb65a9db
automated update of v1.3.0-rc.2 to v1.3.0-rc.3 consensus spec URLs (#4647) 2023-02-21 16:43:21 +00:00
tersec 79eddcde40
capella builder API support (#4643)
* capella builder API support

* use capella EPH when appropriate

* fill in ExecutionPayload.withdrawals and sanity-check builder API withdrawals root
2023-02-21 14:21:38 +01:00
tersec 68cb9fe7b1
use consensus spec v1.3.0-rc.3 test vectors (#4645) 2023-02-21 12:58:08 +00:00
Jacek Sieka 83f9745df1
restore doppelganger check on connectivity loss (#4616)
* restore doppelganger check on connectivity loss

https://github.com/status-im/nimbus-eth2/pull/4398 introduced a
regression in functionality where doppelganger detection would not be
rerun during connectivity loss. This PR reintroduces this check and
makes some adjustments to the implementation to simplify the code flow
for both BN and VC.

* track when check was last performed for each validator (to deal with
late-added validators)
* track when we performed a doppel-detectable activity (attesting) so as
to avoid false positives
* remove nodeStart special case (this should be treated the same as
adding a validator dynamically just after startup)

* allow sync committee duties in doppelganger period

* don't trigger doppelganger when registering duties

* fix crash when expected index response is missing

* fix missing slashingSafe propagation
2023-02-20 13:28:56 +02:00
tersec b19f7001aa
log expected_withdrawals discrepancy (#4595)
* log expected_withdrawals discrepancy

* capella-only rather than not-bellatrix

* add one more (in theory unnecessary, but) guard against get(none)
2023-02-20 09:46:37 +01:00
tersec 72797cad09
switch some beacon_chain/spec/ from eip4844.Foo to deneb.Foo (#4642) 2023-02-20 09:45:49 +01:00
tersec 629b005c27
refactor batch validation not to require genesis_validators_root each time (#4640) 2023-02-20 09:26:22 +01:00
tersec 0cc0c7e6b7
builder spec v0.3.0 URL updates (#4641)
* builder spec v0.3.0 URL updates

* remove stray non-URL/comment change
2023-02-18 01:54:30 +01:00
tersec a382498cfe
batch-verify BLS to execution change messages (#4637) 2023-02-17 13:35:12 +00:00
tersec dc0bbe3a57
rm blockForkAtEpoch and switch callers to consensusForkAtEpoch (#4634) 2023-02-16 21:16:54 +01:00
Zahary Karadjov 8202d4a84e
Version 23.2.0 2023-02-16 20:19:22 +02:00
Eugene Kabanov e91415662b
Keystore cache implementation. (#4372) 2023-02-16 19:25:48 +02:00
Jacek Sieka d63179ab57
vc: avoid npe for unknown validator (#4632)
follow-up for #4590
2023-02-16 13:27:10 +00:00
tersec e342fdd97a
rename stateForkAtEpoch to consensusForkAtEpoch (#4627) 2023-02-16 11:32:12 +02:00
zah ff464e49cf
Implement the set of gas_limit end-points in the Keymanager API (#4612)
Fixes #3946
2023-02-15 15:10:31 +00:00
zah 067ba13c52
Capella metadata for Sepolia (#4615)
Other changes:

Renamed the `EIP_4844_FORK_*` config constants to `DENEB_FORK_*` as
this matches the latest spec and it's already used in the official
Sepolia config.
2023-02-15 14:44:09 +00:00
tersec b7c3037f01
consensus spec ref URL updates to v1.3.0-rc.2 (#4626) 2023-02-15 14:43:51 +00:00
Eugene Kabanov 218ea42220
VC: Fix getBlockRoot() response with execution_optimistic. (#4622)
* Fix getStateRoot() and getBlockRoot() API functions which should obtain `execution_optimistic` field.
Fix sync committee service to check `execution_optimistic` field of getBlockRoot() response.

* 2nd part.

* Remove presets usage.
2023-02-15 15:09:31 +01:00
Etan Kissling 1ac7f1a47a
rm unused `lowParticipation` flag (#4624)
Remove an unused flag and associated (unreachable) logic, as the flag
wasn't even passed to that module.
2023-02-15 00:25:22 +01:00
tersec cf551f10c4
don't fcU on blocks for which block processor received no newPayload reply (#4623) 2023-02-14 21:41:49 +01:00
Etan Kissling 32f6309f95
pass correct CI test flag to initializer (#4621)
Use correct `experimental` flag for running tests.
2023-02-14 14:51:24 +01:00
tersec 3977f1529a
fill in remaining capellaImplementationMissing holes for builder API (#4606) 2023-02-14 11:49:48 +01:00
tersec 3011d49946
refactor fcU sending and rename EL-side root to hash (#4614) 2023-02-14 07:48:39 +01:00
tersec 07ccd3fa6e
remove capella and deneb empty execution payload fallbacks (#4613) 2023-02-13 19:15:16 +02:00
tersec aee19fec6b
block on forkchoiceUpdated EL calls due to doing fewer of them (#4609) 2023-02-13 12:13:52 +01:00
tersec d838e5b6ba
remove unused phase0/altair-specific REST deserialization (#4605) 2023-02-13 12:03:38 +01:00
tersec f9022618f0
deprecate req/resp GetMetaData v1 (#4608) 2023-02-13 12:02:20 +01:00
henridf 59e41dc65d
EIP4844 sync (#4581)
* EIP4844 Sync

* Pass eip4844 fork epoch rather than cfg to syncmanager

* Fix sync

* Update test

* map->mapIt
2023-02-11 20:48:35 +00:00
Etan Kissling 09dd64df32
add deployment phase for Capella-ready networks (#4607)
Allow distinguishing Capella-ready networks from non-upgraded networks
based on `CAPELLA_FORK_EPOCH` being set (Zhejiang public testnet).
2023-02-10 14:26:01 +01:00
tersec 20fc355ff1
final batch of automated 1.3.0-rc.2 consensus-spec urls (#4604) 2023-02-09 22:08:43 +00:00
Jacek Sieka 856fcea8d7
fix slow checking of unknown validators (#4590)
We do a linear scan of all pubkeys for each validator and slot - this
becomes expensive with large validator counts.

* normalise BN/VC validator startup logging
* fix crash when host cannot be resolved while adding remote validator
* silence repeated log spam for unknown validators
* print pubkey/index/activation mapping on startup/validator
identification
2023-02-07 14:53:36 +00:00
Jacek Sieka 98db005c70
seed sync committee state cache (#4592)
By pre-seeding the sync committee cache when applying blocks, we avoid a
significantly expensive validator set traversal / sync committee index
construction during sync / block application - 20-30% sync speedup
post-altair.

* also cache/reload total active balance for another cool 10%
2023-02-07 11:22:22 +01:00
tersec ebb45e10da
initial implementation of capella MEV (#4567) 2023-02-06 19:07:30 +01:00
Jacek Sieka f3ddea6c86
Skip execution payload verification for finalized blocks (#4591)
While syncing the finalized portion of the chain, the execution client
cannot efficiently sync and most of the time returns `SYNCING` - in this
PR, we use CL-verified optmistic sync as long as the block is claimed to
be finalized, only occasionally updating the EL with progress.

Although a peer might lie about what is finalized and what isn't,
eventually we'll call the execution client - thus, all a dishonest
client can do is delay execution verification slightly. Gossip blocks in
particular are never assumed to be finalized.
2023-02-06 08:22:08 +01:00
henridf 9233a52e4b
Don't pass RuntimeConfig to RequestManager (#4589) 2023-02-04 12:13:09 +01:00
tersec bca781b1b0
prioritize REST API-provided BLS to execution changes over gossip-received changes (#4580) 2023-02-03 16:28:28 +01:00
tersec 1c62a5eb24
capella VC support (#4586) 2023-02-03 16:12:11 +01:00
tersec 956aee2d35
fill some capella/EIP4844 missing implementations (#4585) 2023-02-02 22:24:06 +00:00
tersec eb56762d41
some proc -> func in signatures_batch (#4582) 2023-02-02 19:05:30 +00:00
tersec 63ed5885ab
update engine API URLs to v1.0.0-beta.2 (#4579) 2023-02-01 18:49:36 +00:00
tersec bcc9781cc7
rm obsolete interop module (#4570) 2023-02-01 16:29:55 +01:00
henridf 94837caa2a
Eip4844 sync fixes (#4577)
* fixes

* Make some log messages blob-aware

* remove redundant optBlobs()
2023-02-01 14:14:50 +00:00
henridf d7316ac863
Wire up blob fetching with request manager (#4527)
* Wire up blob fetching and request manager

* Update beacon_chain/sync/request_manager.nim

Co-authored-by: tersec <tersec@users.noreply.github.com>

* Review feedback: remove redundant RequestManager field.

* Fix syntax

* Improve blob fetching logic

* fix previous commit

* rman.isBlobsTime(): Remove spurious parameter

* Review feedback

* expressionify an if

---------

Co-authored-by: tersec <tersec@users.noreply.github.com>
2023-02-01 00:25:08 +01:00
tersec cdf69b9360
more automated v1.3.0-rc.1 to v1.3.0-rc.2 consensus spec URL updates (#4572) 2023-01-31 22:49:17 +00:00
Etan Kissling ea6a6b1acd
track slot as part of fork choice debug API (#4565)
Extends fork choice state to also track slot numbers to improve accuracy
of `/eth/v1/debug/fork_choice` endpoint. Autoenable this API on devnet,
and disable some extra checks on devnet to aid focused testing efforts.
Align fork choice pruning logic with API based on checkpoints vs root.
2023-01-31 12:35:01 +00:00
tersec 58ed9308d2
automated v1.3.0-rc.1 to v1.3.0-rc.2 consensus spec URL updates (#4568) 2023-01-31 00:26:57 +01:00
tersec f5e302f973
use 1.3.0-rc.2-hotfix consensus spec test vectors (#4566) 2023-01-30 22:54:16 +00:00
tersec b2f415cbe5
remove merge cruft (#4559) 2023-01-30 20:22:05 +01:00
tersec 8e2792bdd7
remove Nim 1.2 workarounds and `TODO`s (#4533)
* clean up some Nim 1.2 workarounds

* re-add notes about JS backend

* another proc/noSideEffect -> func

* revert ncli/ncli_common.nim changes; 19969 evidently wasn't backported to 1.6
2023-01-30 20:21:51 +01:00
tersec 29ea42ce1a
use v1.3.0-rc.2 consensus spec test vectors (#4564) 2023-01-29 11:28:04 +00:00
tersec 0fb726c420
`BeaconStateFork/BeaconBlockFork` -> `ConsensusFork` (#4560)
* `BeaconStateFork/BeaconBlockFork` -> `ConsensusFork`

* revert unrelated change

* revert unrelated changes

* update test summaries
2023-01-28 19:53:41 +00:00
henridf 67e2f854d5
Fix BeaconBlockAndBlobsSidecar encoding (#4562) 2023-01-28 12:30:38 +00:00
tersec 7635b47a4e
update/clarify Nim issue link after retesting/re-isolating 2023-01-27 10:41:19 +00:00
Etan Kissling 9f279e2933
cache LC headers and sync committees for bootstrap (#4499)
To allow LC data retention longer than the one for historic states,
introduce persistent DB caches for `current_sync_committee` and
`LightClientHeader` for finalized epoch boundary blocks.
This way, historic `LightClientBootstrap` requests may still be honored
even after pruning. Note that historic `LightClientUpdate` requests are
already answered using fully persisted objects, so don't need changes.

Sync committees and headers are cached on finalization of new data.
For existing data, info is lazily cached on first access.

Co-authored-by: Jacek Sieka <jacek@status.im>
2023-01-27 10:44:57 +01:00
Zahary Karadjov daa99f43c3
Merge branch 'stable' into unstable 2023-01-26 16:47:26 +01:00
Eugene Kabanov 08ed8ad43e
Adopt BN and VC header sizes and requirements, to avoid users confusion with default configuration options. (#4556)
Add comments.
2023-01-26 16:00:10 +01:00
tersec c22384d7ba
always use Bellatrix message size limits (#4558) 2023-01-26 14:24:23 +00:00
Zahary Karadjov 1ea102b033
Version 23.1.1 2023-01-26 14:37:03 +01:00
Jacek Sieka 0c1686eb20
fix pruned backfill summary (fixes #4552)
When enabling pruned history mode, the backfill summary as it was
computed with full backfilling may be pruned - don't try to load it on
init
2023-01-26 14:23:12 +01:00
Jacek Sieka fa27ce76a3
vmon: set details mode to default off
The "on" default for validator monitor details incurs a heavy
performance penalty on large-validator setups - this may cause excess
memory usage or slowdowns when metrics are queried - this PR changes the
default to off, as was intended for the 23.1.0 release.
2023-01-26 14:18:52 +01:00
Jacek Sieka 968e27bc97
fix pruned backfill summary (fixes #4552) (#4554)
When enabling pruned history mode, the backfill summary as it was
computed with full backfilling may be pruned - don't try to load it on
init
2023-01-26 11:23:17 +01:00
henridf 03b468c537
Pass correct block root to validate_blobs_sidecar (#4557) 2023-01-26 09:25:37 +00:00
henridf 7966ab6be2
Some EIP4844 fixes (#4549)
* debug log upon sidecar validation failure

* Fill in signature catch upon SignedBeaconBlockAndBlobsSidecar deser

* Always fill blobssidecar slot and root

* Skip lastFCU when eth1monitor is nil

* fix

* Use cached root
2023-01-25 18:35:46 +01:00