Commit Graph

5049 Commits

Author SHA1 Message Date
tersec b975455254
avoid gnosis-build-triggered OOMs (#4531) 2023-01-20 14:14:08 +00:00
Etan Kissling a5596dc5e6
add documentation for `shouldSyncOptimistically` (#4530)
Clarify rationale behind the `shouldSyncOptimistically` workaround for
why we subscribe to blocks gossip when light client syncing.
2023-01-20 05:31:05 +01:00
tersec cd58613615
remaining automated rc.0 to rc.1 consensus spec ref URL changes (#4529) 2023-01-20 00:20:51 +00:00
Etan Kissling 65ca523482
support EIP4844 LC data format (#4520)
Implements the proposed light client data format for EIP4844:
- https://github.com/ethereum/consensus-specs/pull/3151
2023-01-19 23:53:37 +01:00
tersec 819e007689
exit/validatorchange pool includes BLS to execution messages; REST support for new pool (#4519)
* exit/validatorchange pool includes BLS to execution messages; REST
support for new pool

* catch failed individual futures

* increase BLS changes bound and keep BLS seen consistent with subpool

* deque capacities should be powers of 2
2023-01-19 22:00:40 +00:00
tersec 6818120849
drop Nim 1.2 testing from GitHub Actions CI (#4525) 2023-01-19 17:55:39 +00:00
Marius Kjærstad b2b111b474
Changed http:// to https:// on some links (#4526)
Changed http:// to https:// on some links in README.md
2023-01-19 15:01:14 +00:00
zah ae301af291
Fix #4500 (#4523) 2023-01-18 19:20:28 +02:00
zah a2bc10e51c
Makefile targets for capella-devnet-3 (#4521) 2023-01-18 17:19:02 +02:00
Etan Kissling 58f4c13e14
support Capella LC data format (#4514)
Implements the proposed light client data format for Capella:
- https://github.com/ethereum/consensus-specs/pull/3151
2023-01-18 15:32:57 +01:00
tersec aea7a0c8b8
remove TTD monitoring (#4486) 2023-01-18 16:01:49 +02:00
Etan Kissling 9cceb1b4a0
support `readOnly` backend in LC client/data dbs (#4518)
When accessing DB in `readOnly` mode that does not already have latest
schema, initial writes trigger `attempt to write a readonly database`.
Avoid that by only writing schema when DB is not `readOnly`, and provide
data from legacy tables if such are present.
2023-01-18 03:06:23 +01:00
tersec 073c544f0c
automated update from v1.3.0-rc.0 to v1.3.0-rc.1 consensus spec URLs (#4517) 2023-01-17 16:10:52 +00:00
tersec 9b0868033e
use GitHub Actions cache and upload-artifact v3 (#4516) 2023-01-17 15:45:04 +00:00
henridf 05a6c6280d
Add `beacon_block_and_blobs_sidecar_by_root` RPC (#4475)
* Implement beacon_block_and_blobs_sidecar_by_root

* review feedback

* more review feedback
2023-01-16 17:08:16 +00:00
henridf 727920a571
Refactor block/blobs types (#4491)
* Refactor block/blobs types

Use type system to enforce invariant that a pre-4844 block cannot have
a sidecar.

* Update beacon_chain/nimbus_beacon_node.nim

Co-authored-by: tersec <tersec@users.noreply.github.com>

* review feedback

Co-authored-by: tersec <tersec@users.noreply.github.com>
2023-01-16 16:26:48 +00:00
Etan Kissling fda03548e3
use `ForkedLightClientStore` internally (#4512)
When running `nimbus_light_client`, we persist the latest header from
`LightClientStore.finalized_header` in a database across restarts.
Because the data format is derived from the latest `LightClientStore`,
this could lead to data being persisted in pre-release formats.

To enable us to test later `LightClientStore` versions on devnets,
transition to a `ForkedLightClientStore` internally that is only
migrated to newer forks on-demand (instead of starting at latest).
2023-01-16 16:53:45 +01:00
tersec 87a34bff6c
build largest two RAM consumers, nimbus_beacon_node and all_tests, by default separately (#4513) 2023-01-16 11:34:40 +01:00
Jacek Sieka d8caab500d
Enable validator monitor by default (#4468)
By enabling the validator monitor, more precise information about the
lifecycle of an attestation is logged at the higher `NOTICE` log level
while current `sent` messages are logged at `INF` instead, since they
are less interesting.

In particular, missed attestations and those that vote for the wrong
head are now detected and logged at NOTICE.

In addition to logging, this feature enables rich metrics around
attestation and sync committee performance - by default, validators are
tracked in aggregate but a detailed mode exists as well

This feature has been available since early Nimbus days, but it has now
been tuned and optimised such that it is safe to enable by default, even
for large setups.

* enable automatic validator monitoring by default
* replace `--validator-monitor-totals` flag with
`--validator-monitor-details` - the detailed mode is disabled by default
* lower "sent" log level to `INF` for several messages - in particular
those that are traced by the validator monitor

This is a retake on #3531 which was later reverted in #3578.
2023-01-16 11:28:35 +01:00
tersec 7ad0d3e6c2
attempt to reduce GitHub Actions runner disk usage (#4511) 2023-01-15 08:43:50 +00:00
tersec 6ccf13e762
log Windows GitHub Action runner partition sizes (#4510) 2023-01-14 21:48:01 +00:00
Etan Kissling 609227559f
LC data fork cleanup (#4506)
Distinguish between those code locations that need to be updated on each
light client data format change, and those others that should generally
be fine, as long as a valid light client object is processed.

The former are tagged with static assert for `LightClientDataFork.high`.

The latter are changed to `lcDataFork > LightClientDataFork.None` to
indicate that they depend only on presence of any valid object.
Also bundled a few minor cleanups and fixes.

Also add `Forky` type for `LightClientStore` and minor fixes / cleanups.
2023-01-14 22:19:50 +01:00
tersec eb02415f00
execution/engine withdrawals amount in uint64 gwei (#4509) 2023-01-14 17:26:57 +00:00
tersec 68e08c6f15
use v1.3.0-rc.1 consensus spec test vectors (#4505) 2023-01-14 04:20:51 +00:00
tersec d2e9cdf1a0
show number of cores and total physical memory on Windows GitHub Actions (#4504) 2023-01-13 21:42:08 +00:00
Etan Kissling 2324136552
add `LightClientHeader` wrapper (#4481)
The light client data structures were changed to accommodate additional
fields in future forks (e.g., to also hold execution data).

There is a minor change to the JSON serialization, where the `header`
properties are now nested inside a `LightClientHeader`.
The SSZ serialization remains compatible.

See https://github.com/ethereum/consensus-specs/pull/3190
and https://github.com/ethereum/beacon-APIs/pull/287
2023-01-13 16:46:35 +01:00
henridf 77530841e3
Fix eip4844 gossip transition (#4498)
* Fix eip4844 gossip transition

* Future-proof block topic selection

* whitespace
2023-01-13 15:37:31 +00:00
tersec e09904adab
Revert "reduce intermittent Windows GitHub Actions CI build failures (#4495)" (#4502)
This reverts commit 3b60b225b3.
2023-01-13 13:22:35 +00:00
Zahary Karadjov 46fc5716a4
Faster keystore generation in the local sim 2023-01-13 12:31:39 +02:00
zah 0f758c5f02
Working Makefile targets for Capella devnet2 (#4494)
* Working Makefile targets for Capella devnet2

make capella-devnet-2
make clean-capella-devnet-2

You'll need to have https://github.com/tmuxinator/tmuxinator installed.
It's available as a regular package in most Linux distributions or through
Nix or Brew on macOS.

This commit also fixes the initial hang in the Eth1 monitor in the "find
TTD block" procedure through a fix to the network metadata files which
hasn't been upstreamed yet.

Other changes:

* Disabled Geth snap sync in the simulation

When all Geth nodes are configured to run with snap sync enabled, they all
start snap sync after the first forkchoiceUpdated which causes the BNs to
skip validator duties because the EL is syncing. The snap sync never completes
due to poor connectivity between the Geth nodes in the simulation.
2023-01-13 12:21:58 +02:00
zah 9861f0ab7d
Enable the gnosis build in CI on non-Windows targets (#4501) 2023-01-13 12:17:53 +02:00
Zahary Karadjov 3b03ef8ffb
Remove the genesis detection code 2023-01-13 04:28:30 +02:00
Zahary Karadjov 2ac28b1346
Disable the CI build of Gnosis 2023-01-13 04:28:30 +02:00
Zahary Karadjov b06502bf65
Gnosis const preset 2023-01-13 04:28:29 +02:00
Zahary Karadjov c01e53e35e
Fix bitrot in the gnosis build; Up-to-date bootstrap nodes
To prevent similar bitrot from occuring in the future, the gnosis-build
is now part of the default `make` targets.
2023-01-13 04:28:29 +02:00
henridf 9da64bb605
Far-future epoch check before installing 4844 vals (#4497) 2023-01-12 21:46:12 +01:00
Etan Kissling 7e276937dc
make LC data fork aware (#4493)
In a future fork, light client data will be extended with execution info
to support more use cases. To anticipate such an upgrade, introduce
`Forky` and `Forked` types, and ready the database schema.
Because the mapping of sync committee periods to fork versions is not
necessarily unique (fork schedule not in sync with period boundaries),
an additional column is added to `period` -> `LightClientUpdate` table.
2023-01-12 18:11:38 +01:00
tersec 3b60b225b3
reduce intermittent Windows GitHub Actions CI build failures (#4495) 2023-01-12 14:47:33 +00:00
Jacek Sieka 6bfc766629
drop subset sync contributions in gossip (#4490)
* correctly report ignored contributions in metrics
* avoid counting subset contributions in vmon (bring in line with
attestation aggregates)
* avoid signature checks for subset attestations

A being a non-strict subset is a sufficient condition to ignore.
2023-01-12 15:08:08 +01:00
tersec 4f9064a153
consensus spec ref URL updates (#4496) 2023-01-12 13:33:14 +00:00
henridf 5a03e7400e
Blobs sidecars by range (#4464)
* Add blobs_sidecars_by_range RPC

* Update beacon_chain/spec/datatypes/eip4844.nim

Co-authored-by: tersec <tersec@users.noreply.github.com>

Co-authored-by: tersec <tersec@users.noreply.github.com>
2023-01-12 12:33:24 +00:00
henridf 309f8690de
Wire up engine_newPayloadV3 (#4482)
* Wire up eip4844's newPayloadV3

* Add eip4844 test

* Update AllTests-mainnet.md and fix typo
2023-01-11 18:21:19 +00:00
tersec b14106b72b
preserve historical_summaries in capella to eip4844 fork (#4492) 2023-01-11 17:21:16 +00:00
Jacek Sieka 3e565e9878
exportEra: allow exporting pruned databases (#4485)
When a database has been pruned, we can still export the non-pruned part
- running the era exported together with pruning allows archiving the
full ethereum history for future reference without wasting space in the
database.

* use logging for reporting era write progress
* less noise when skipping existing files
* load blocks from era store also when working with `ncli_db`
* write to temporary file then rename when era is complete, to reduce
risk of corruption
* also avoids loading the in-progress era file when writing and
reading from the same era folder
2023-01-11 17:20:47 +01:00
Jacek Sieka ba3db7aa5a
spec: Option -> Opt (#4488) 2023-01-11 12:29:21 +00:00
Zahary Karadjov 713bdd317d
Bugfix: Handle networks with TTD=0 (e.g. withdrawal devnets) 2023-01-10 19:17:11 +02:00
tersec e28e1aeec8
a few consensus spec ref URL updates (#4483) 2023-01-10 16:14:17 +00:00
Etan Kissling ca01356d8e
stricter check for context bytes when recv LC data (#4480)
Bellatrix and Altair light client data share same body, but have other
fork digests. Validate that the peer's sent fork digest matches the one
that we expect (derived from `attested_header.slot`).
2023-01-10 16:54:29 +01:00
tersec 2fc0fa4288
don't use non-engine-API-whitelisted web3 methods (#4484) 2023-01-10 15:50:34 +00:00
Etan Kissling 0590be7afe
add `num_active_participants` helpers for `SyncAggregate` (#4478)
Introduce `num_active_participants` helper function to reduce visibility
of low-level `countOnes` function and reduce code duplication.
2023-01-10 11:26:25 +00:00