Commit Graph

29 Commits

Author SHA1 Message Date
Etan Kissling 508819633a
more consensus-specs v1.3.0 bumps (#4898)
Updates more consensus-specs references to v1.3.0.
This PR only includes updates for sections otherwise unmodified.
2023-05-05 20:38:51 +00:00
Etan Kissling c4e258af2a
check signatures in CP sync backfill with LC (#4858)
When using trusted node sync with light client (`--trusted-block-root`),
the trust assumption on the server is reduced to solely be responsible
for data availability, but not data correctness. This means that we must
check block proposer signatures against the downloaded checkpoint, as
they are not covered by the block root.

Note that this lowers the backfill speed when using LC based CP sync
due to the extra checks, by about 60% for me.
2023-04-26 13:37:27 +02:00
Jacek Sieka 7df75d77fa
move spec constants to spec (#4825)
* move spec constants to spec

quite funny, this removes >_1000_ transitive imports when building
`nimbus_validator_client`.

`genDepend`:

```sh
[arnetheduck@praeceps beacon_chain]$ diff -u
nimbus_validator_client.dot.pre nimbus_validator_client.dot | wc
   1167    3507  103822
```

* imports

* better constants

* oops

* too many blobs

* reorder spec constants in fork order

* cleanup
2023-04-17 17:18:54 +00:00
Etan Kissling cb9e0eed49
allow trusted node sync based on LC trusted block root (#4736)
* allow trusted node sync based on LC trusted block root

Extends `trustedNodeSync` with a new `--trusted-block-root` option that
allows initializing a light client. No `--state-id` must be provided.
The beacon node will then use this light client to obtain the latest
finalized state from the remote server in a trust-minimized fashion.
Note that the provided `--trusted-block-root` should be somewhat recent,
and that security precautions such as comparing the state root against
block explorers is still recommended.

* fix

* workaround for `valueOr` limitations

* reduce magic numbers

* digest len > context len for readability

* move `cstring` conversion to caller

* avoid abbreviations

* `return` codestyle
2023-04-16 06:07:07 +00:00
Etan Kissling ad118cd354
rename `stateFork` > `consensusFork` (#4718)
Just the variable, not yet `lcDataForkAtStateFork` / `atStateFork`.

- Shorten comment in `light_client.nim` to keep line width
- Do not rename `stateFork` mention in `runProposalForkchoiceUpdated`.
- Do not rename `stateFork` in `getStateField(dag.headState, fork)`

Rest is just a mechanical mass replace
2023-03-11 00:35:52 +00:00
tersec a47f0b054e
finish eip4844 to deneb module rename (#4705) 2023-03-09 01:34:17 +01:00
tersec e342fdd97a
rename stateForkAtEpoch to consensusForkAtEpoch (#4627) 2023-02-16 11:32:12 +02:00
tersec aacc8d702d
remove Nim 1.2-compatible `push raise`s and update copyright notice years (#4528) 2023-01-20 14:14:37 +00:00
henridf 64878888bd
Blob storage (#4454)
* Blob storage

* fix indentation

* Fix build (none->Opt.none)

* putBlobs -> putBlobsSidecar

* getBlobs -> getBlobsSidecar

* Check blob correctness when storing a backfill block

* Blobs table: rename and conditionally create

* Check block<->blob match in storeBackfillBlock

* Use when .. toFork() to condition on type

* Check blob viability in block_processor.storeBlock()

* Fix build

* Review feedback
2023-01-09 18:42:10 +00:00
Jacek Sieka 75c7195bfd
Backfill only up to MIN_EPOCHS_FOR_BLOCK_REQUESTS blocks (#4421)
When backfilling, we only need to download blocks that are newer than
MIN_EPOCHS_FOR_BLOCK_REQUESTS - the rest cannot reliably be fetched from
the network and does not have to be provided to others.

This change affects only trusted-node-synced clients - genesis sync
continues to work as before (because it needs to construct a state by
building it from genesis).

Those wishing to complete a backfill should do so with era files
instead.
2022-12-23 08:42:55 +01:00
tersec 7faef7827e
fix EIP4844 withBlck (#4411)
* fix EIP4844 withBlck

* don't raiseAssert by default
2022-12-14 18:30:56 +01:00
zah d30cb8baf1
Support for obtaining deposit snapshots during trustedNodeSync (#4303)
Other changes:

* More optimal search for TTD block.

* Add timeouts to all REST requests during trusted node sync.
  Fixes #4037

* Removed support for storing a deposit snapshot in the network
  metadata.
2022-12-07 12:24:51 +02:00
Etan Kissling 48994f67d3
rename `BlockError` -> `VerifierError` (#4310)
We currently use `BlockError` for both beacon blocks and LC objects.
In light of EIP4844, we will likely also use it for blob sidecars.
To avoid confusion, renaming it to a more generic `VerifierError`,
and update its documentation to be more generic.

To avoid long lines as a followup, also renaming the `block_processor`'s
`BlockProcessingCompleted.completed`->`ProcessingStatus.completed` and
`BlockProcessingCompleted.notCompleted`->`ProcessingStatus.notCompleted`
2022-11-10 17:40:27 +00:00
Jacek Sieka 09ade6d33d
Make trusted node sync era-aware (#4283)
This PR removes a bunch of code to make TNS aware of era files, avoiding
a duplicated backfill when era files are available.

* reuse chaindag for loading backfill state, replacing the TNS homebrew
* fix era block iteration to skip empty slots
* add tests for `can_advance_slots`
2022-11-10 10:44:47 +00:00
Jacek Sieka d839b9d07e
State-only checkpoint state startup (#4251)
Currently, we require genesis and a checkpoint block and state to start
from an arbitrary slot - this PR relaxes this requirement so that we can
start with a state alone.

The current trusted-node-sync algorithm works by first downloading
blocks until we find an epoch aligned non-empty slot, then downloads the
state via slot.

However, current
[proposals](https://github.com/ethereum/beacon-APIs/pull/226) for
checkpointing prefer finalized state as
the main reference - this allows more simple access control and caching
on the server side - in particular, this should help checkpoint-syncing
from sources that have a fast `finalized` state download (like infura
and teku) but are slow when accessing state via slot.

Earlier versions of Nimbus will not be able to read databases created
without a checkpoint block and genesis. In most cases, backfilling makes
the database compatible except where genesis is also missing (custom
networks).

* backfill checkpoint block from libp2p instead of checkpoint source,
when doing trusted node sync
* allow starting the client without genesis / checkpoint block
* perform epoch start slot lookahead when loading tail state, so as to
deal with the case where the epoch start slot does not have a block
* replace `--blockId` with `--state-id` in TNS command line
* when replaying, also look at the parent of the last-known-block (even
if we don't have the parent block data, we can still replay from a
"parent" state) - in particular, this clears the way for implementing
state pruning
* deprecate `--finalized-checkpoint-block` option (no longer needed)
2022-11-02 10:02:38 +00:00
Jacek Sieka af9ec577d0
nicer error message for failed backfill (#4188)
* nicer error message for failed backfill

Many checkpoint sources don't support block download

* RestGenericError -> RestErrorMessage

...and other assorted fixes to bring rest types closer to spec

* fix tests
2022-09-29 23:55:18 +03:00
tersec 02a99543c6
more `withState` `state` -> `forkyState` (#4112) 2022-09-13 14:53:12 +03:00
Jacek Sieka 91a1b4e0c5
better error message on invalid URL (fixes #4023) (#4024) 2022-08-26 15:47:55 +00:00
Zahary Karadjov 7f681af56f
Make it easier to test trustedNodeSync or different networks 2022-08-18 19:48:11 +03:00
Taneli Hukkinen 17bf42316e
Fix typo in log: canoncial->canonical (#3927) 2022-08-01 00:32:08 +00:00
Miran dfd4afc9f2
compatibility with Nim 1.4+ (#3888) 2022-07-29 10:53:42 +00:00
tersec 56cd6e2b45
rm unused imports and fix stylecheck issues (#3784) 2022-06-21 21:01:45 +02:00
tersec 8eb5d5de09
use ZERO_HASH for default(Eth2Digest)/Eth2Digest() in func calls (#3770) 2022-06-18 04:57:37 +00:00
Jacek Sieka b3d80827fb
tns: checkpoint wal periodically while backfilling (#3516)
Witout this, we end up with a massive .wal file that needs to be
checkpointed on first startup (which takes a few minutes) - it's much
more efficient to do smaller checkpoints, it turns out.
2022-03-18 12:32:20 +01:00
Jacek Sieka d0183ccd77
Historical state reindex for trusted node sync (#3452)
When performing trusted node sync, historical access is limited to
states after the checkpoint.

Reindexing restores full historical access by replaying historical
blocks against the state and storing snapshots in the database.

The process can be initiated or resumed at any point in time.
2022-03-11 12:49:47 +00:00
Jacek Sieka 1f89b7f7b9
speed up trusted node backfill (#3371)
With these changes, we can backfill about 400-500 slots/sec, which means
a full backfill of mainnet takes about 2-3h.

However, the CPU is not saturated - neither in server nor in client
meaning that somewhere, there's an artificial inefficiency in the
communication - 16 parallel downloads *should* saturate the CPU.

One plasible cause would be "too many async event loop iterations" per
block request, which would introduce multiple "sleep-like" delays along
the way.

I can push the speed up to 800 slots/sec by increasing parallel
downloads even further, but going after the root cause of the slowness
would be better.

* avoid some unnecessary block copies
* double parallel requests
2022-02-12 12:09:59 +01:00
Jacek Sieka d076e1a11b
ncli_db: import states and blocks from era file (#3313) 2022-01-25 09:28:26 +01:00
Jacek Sieka 61342c2449
limit by-root requests to non-finalized blocks (#3293)
* limit by-root requests to non-finalized blocks

Presently, we keep a mapping from block root to `BlockRef` in memory -
this has simplified reasoning about the dag, but is not sustainable with
the chain growing.

We can distinguish between two cases where by-root access is useful:

* unfinalized blocks - this is where the beacon chain is operating
generally, by validating incoming data as interesting for future fork
choice decisions - bounded by the length of the unfinalized period
* finalized blocks - historical access in the REST API etc - no bounds,
really

In this PR, we limit the by-root block index to the first use case:
finalized chain data can more efficiently be addressed by slot number.

Future work includes:

* limiting the `BlockRef` horizon in general - each instance is 40
bytes+overhead which adds up - this needs further refactoring to deal
with the tail vs state problem
* persisting the finalized slot-to-hash index - this one also keeps
growing unbounded (albeit slowly)

Anyway, this PR easily shaves ~128mb of memory usage at the time of
writing.

* No longer honor `BeaconBlocksByRoot` requests outside of the
non-finalized period - previously, Nimbus would generously return any
block through this libp2p request - per the spec, finalized blocks
should be fetched via `BeaconBlocksByRange` instead.
* return `Opt[BlockRef]` instead of `nil` when blocks can't be found -
this becomes a lot more common now and thus deserves more attention
* `dag.blocks` -> `dag.forkBlocks` - this index only carries unfinalized
blocks from now - `finalizedBlocks` covers the other `BlockRef`
instances
* in backfill, verify that the last backfilled block leads back to
genesis, or panic
* add backfill timings to log
* fix missing check that `BlockRef` block can be fetched with
`getForkedBlock` reliably
* shortcut doppelganger check when feature is not enabled
* in REST/JSON-RPC, fetch blocks without involving `BlockRef`

* fix dag.blocks ref
2022-01-21 13:33:16 +02:00
Jacek Sieka 68247f81b3
Trusted node sync (#3209)
* Trusted node sync

Trusted node sync, aka checkpoint sync, allows syncing tyhe chain from a
trusted node instead of relying on a full sync from genesis.

Features include:

* sync from any slot, including the latest finalized slot
* backfill blocks either from the REST api (default) or p2p (#3263)

Future improvements:

* top up blocks between head in database and some other node - this
makes for an efficient backup tool
* recreate historical state to enable historical queries

* fixes

* load genesis from network metadata
* check checkpoint block root against state
* fix invalid block root in rest json decoding
* odds and ends

* retry looking for epoch-boundary checkpoint blocks
2022-01-17 10:27:08 +01:00