Commit Graph

4505 Commits

Author SHA1 Message Date
tersec a18b39c9c1
JWT support (#3561) 2022-03-31 14:43:05 +00:00
zah f89c604fb0
Add Nimble lock file (#3560) 2022-03-30 22:00:03 +00:00
Kim De Mey e222405b86
Bump nim-eth vendor module (#3556)
Changes in nim-eth relevant to nimbus-eth2:

- Debug logs providing info for the initial discoveryv5 configuration
- net/nat.nim: "raises" annotations
2022-03-30 08:30:36 +02:00
Jacek Sieka 5092fc41c7
use snappy-framed format for compressing bellatrix+ database entries (#3551)
`.era` files and Req/Resp protocols use framed formats - aligning the
database with these makes for less recompression work overall as gossip
is sent only once while req/resp repeats (potentially) - this also
allows efficient pruning-to-era where snappy-recompression is the major
cycle thief.
2022-03-29 11:33:06 +00:00
Etan Kissling 2c5f725543
reformat long lines (#3539)
Shortens some long lines by introducing temp variables and line breaks.
2022-03-29 09:15:42 +02:00
Ștefan Talpalaru 46e5175267
Windows: latest Mingw can't handle '\' as path sep (#3553) 2022-03-29 08:39:24 +02:00
tersec 391d4ddd0d
bump faststreams to avoid triggering Nim checks (#3552) 2022-03-28 22:04:43 +00:00
tersec d63b836f61
cap number of threads for --num-threads=0 (#3547)
* cap number of threads for --num-threads=0

* cap at 16 threads instead
2022-03-28 16:33:25 +00:00
Dustin Brody 57fc9d5433
revert 2022-03-28 12:08:42 +00:00
Dustin Brody 146a4c097d
bump faststreams to avoid triggering Nim checks 2022-03-28 12:05:57 +00:00
Ștefan Talpalaru d7d7398fe6
dist: build Windows binary with newer compiler (#3548)
We went from (Mingw-w64) GCC 5.5.0 to 11.2.0. Progress!
2022-03-28 10:03:11 +02:00
tersec 9b43a76f2f
kiln beacon node (#3540)
* kiln bn

* use  version of beacon_chain_db

* have Eth1Monitor abstract more tightly over web3provider
2022-03-25 11:40:10 +00:00
tersec b37bf8c94b
remove TaintedString (#3546)
* remove TaintedString

* dumpDir is used as a helper function for more specific types of dumpDirs
2022-03-24 21:44:34 +00:00
Jacek Sieka e009728858
work around Nim assignment bug that breaks state pruning (#3545)
See https://github.com/nim-lang/Nim/issues/19613
2022-03-24 14:37:37 +00:00
Jacek Sieka 6983dacc26
fix bellatrix table names (#3544)
this should/will cause existing nimbus databases to revert to the altair
merge and resync with the new table name
2022-03-24 14:36:31 +01:00
Dustin Brody 3d9ef9b9b6
clean up action tracker 2022-03-24 10:57:20 +00:00
tersec a058b1d1ce
bump confutils to fix 64-bit cmdline args on 32-bit platforms (#3542) 2022-03-24 09:58:48 +01:00
tersec 50f5754e3c
exists{Dir,File} -> {dir,file}Exists; rm unused imports (#3543) 2022-03-24 00:38:48 +00:00
Jacek Sieka bc4d4e1741
remove redundant `process_block` overloads (#3541) 2022-03-23 12:49:54 +00:00
Jacek Sieka bc80ac3be1
harden REST API `atSlot` against non-finalized blocks (#3538)
* harden validator API against pre-finalized slot requests
* check `syncHorizon` when responding to validator api requests too far
from `head`
* limit state-id based requests to one epoch ahead of `head`
* put historic data bounds on block/attestation/etc validator production API, preventing them from being used with already-finalized slots
* add validator block smoke tests
* make rest test create a new genesis with the tests running roughly in
the first epoch to allow testing a few more boundary conditions
2022-03-23 12:42:16 +01:00
Jacek Sieka 4207b127f9
era: load blocks and states (#3394)
* era: load blocks and states

Era files contain finalized history and can be thought of as an
alternative source for block and state data that allows clients to avoid
syncing this information from the P2P network - the P2P network is then
used to "top up" the client with the most recent data. They can be
freely shared in the community via whatever means (http, torrent, etc)
and serve as a permanent cold store of consensus data (and, after the
merge, execution data) for history buffs and bean counters alike.

This PR gently introduces support for loading blocks and states in two
cases: block requests from rest/p2p and frontfilling when doing
checkpoint sync.

The era files are used as a secondary source if the information is not
found in the database - compared to the database, there are a few key
differences:

* the database stores the block indexed by block root while the era file
indexes by slot - the former is used only in rest, while the latter is
used both by p2p and rest.
* when loading blocks from era files, the root is no longer trivially
available - if it is needed, it must either be computed (slow) or cached
(messy) - the good news is that for p2p requests, it is not needed
* in era files, "framed" snappy encoding is used while in the database
we store unframed snappy - for p2p2 requests, the latter requires
recompression while the former could avoid it
* front-filling is the process of using era files to replace backfilling
- in theory this front-filling could happen from any block and
front-fills with gaps could also be entertained, but our backfilling
algorithm cannot take advantage of this because there's no (simple) way
to tell it to "skip" a range.
* front-filling, as implemented, is a bit slow (10s to load mainnet): we
load the full BeaconState for every era to grab the roots of the blocks
- it would be better to partially load the state - as such, it would
also be good to be able to partially decompress snappy blobs
* lookups from REST via root are served by first looking up a block
summary in the database, then using the slot to load the block data from
the era file - however, there needs to be an option to create the
summary table from era files to fully support historical queries

To test this, `ncli_db` has an era file exporter: the files it creates
should be placed in an `era` folder next to `db` in the data directory.
What's interesting in particular about this setup is that `db` remains
as the source of truth for security purposes - it stores the latest
synced head root which in turn determines where a node "starts" its
consensus participation - the era directory however can be freely shared
between nodes / people without any (significant) security implications,
assuming the era files are consistent / not broken.

There's lots of future improvements to be had:

* we can drop the in-memory `BlockRef` index almost entirely - at this
point, resident memory usage of Nimbus should drop to a cool 500-600 mb
* we could serve era files via REST trivially: this would drop backfill
times to whatever time it takes to download the files - unlike the
current implementation that downloads block by block, downloading an era
at a time almost entirely cuts out request overhead
* we can "reasonably" recreate detailed state history from almost any
point in time, turning an O(slot) process into O(1) effectively - we'll
still need caches and indices to do this with sufficient efficiency for
the rest api, but at least it cuts the whole process down to minutes
instead of hours, for arbitrary points in time

* CI: ignore failures with Nim-1.6 (temporary)

* test fixes

Co-authored-by: Ștefan Talpalaru <stefantalpalaru@yahoo.com>
2022-03-23 09:58:17 +01:00
Etan Kissling 49673c4410
refer to `syncCommitteeMsgPool` consistently (#3537)
Updates a `sync_committee_msg_pool` reference to camelCase.
2022-03-23 07:46:48 +01:00
Etan Kissling b2b7b0bd56
serve libp2p protocol for light client sync (#3341)
This extends the `--serve-light-client-data` launch option to serve
locally collected light client data via libp2p.
Backfill of historic best `LightClientUpdate` is not yet implemented.
See https://github.com/ethereum/consensus-specs/pull/2802
2022-03-22 21:23:36 +01:00
Jacek Sieka 70270eeabe
better error messages on directory creation failure (#3536) 2022-03-22 17:06:21 +00:00
tersec e7d017b50c
remove AppVeyor, Travis, and Azure CI pipeline definitions (#3535) 2022-03-22 11:16:35 +00:00
Jacek Sieka 4a237cb908
enable chronosStrictException (#3533)
* bump nim-json-rpc
2022-03-22 09:42:28 +01:00
Etan Kissling 8cc394ba49
extract DAG dependent init to own function (#3530)
During operation as a light client, the chain DAG is not available.
As a preparation, the beacon node initialization logic is divided into
parts depending on the presence of the chain DAG, and parts that are
always available (including a future light client mode).
This is a pure code move without semantic changes.
2022-03-21 17:52:15 +01:00
Jacek Sieka 361596c719
harden head update against missing parent (#3529)
in case BlockRef ends up in some lifetime leak

* fix duplicate head logging
2022-03-21 15:18:05 +01:00
Jacek Sieka 13fafe3a40
simplify unviable head pruning (#3528)
Also note bug that exists that potentially prevents states from being
pruned correctly
2022-03-21 09:20:26 +00:00
Etan Kissling fd1ffd62dd
update light client server for DAG failure modes (#3514)
Gracefully handles the new failure modes recently introduced to the DAG
as part of https://github.com/status-im/nimbus-eth2/pull/3513
Data that is deemed to exist but fails to load leads to an error log to
avoid suppressing logic errors accidentally. In `verifyFinalization`
mode, the assertions remain active.
2022-03-20 11:58:59 +01:00
Etan Kissling 04b851f775
fix light client data pruning (#3523)
When eliminating orphaned forks, light client data about blocks was also
deleted when the orphaned fork was referring to a state several slots
after the block. Linking light client data pruning with block deletion
instead of state deletion fixes this problem. Light client data always
refers to blocks and their immediate post-state.
2022-03-20 10:09:43 +01:00
Etan Kissling ca045900c8
extract chain DAG loading to separate function (#3527)
When transitioning from light client to full node the chain DAG will be
loaded separately from the rest of the beacon node initialization.
Extracting chain DAG loading to a separate function will allow reusing
a lot of the existing code. This code move doesn't change semantics.
2022-03-19 17:48:24 +01:00
Jacek Sieka ea1acd7397
fix loading when finalized checkpoint slot is missing block (#3525)
ref loop would stop one block early in this case - trying to load
everything in one loop ends up being pretty confusing..

* simplify finalizedBlocks topup by splitting it from the head loop /
query
2022-03-19 11:02:17 +00:00
Jacek Sieka e418497bb2
make attestation duty minimum offset relative to slot length (#3522) 2022-03-19 09:59:13 +01:00
Etan Kissling 637f1e2be6
simplify `computeEarliestLightClientSlot` (#3524)
Combine DAG and LC import tails in `computeEarliestLightClientSlot`.
2022-03-19 09:58:55 +01:00
Ștefan Talpalaru ea5c052016
enable multithreading by default (10-20% faster sync) (#3493) 2022-03-19 08:59:10 +01:00
Ștefan Talpalaru a1f3adc3e2
bump vendor/nimbus-build-system (#3526)
* bump vendor/nimbus-build-system
2022-03-19 08:58:05 +01:00
Etan Kissling 18bd6df1b4
fix light client data collection for checkpoint sync (#3498)
When doing checkpoint sync, collecting light client data of known blocks
and states incorrectly assumes that `finalized_checkpoint` information
is also known. Hardens collection to only collect finalized checkpoint
data after `dag.computeEarliestLightClientSlot`.
2022-03-18 15:47:53 +01:00
Jacek Sieka d0223d1f28
fix finalized epoch ref loading on checkpoint start (#3517)
regression from #3513 that did not take tail into consideration when
loading epoch ancestor
2022-03-18 13:13:57 +01:00
tersec d11d61c745
engine API alpha.7 -> alpha.8 and a few remaining v1.1.9 to v1.1.0 CL spec URL updates (#3519) 2022-03-18 11:46:39 +00:00
Jacek Sieka 0db1e768e4
don't write `node-metadata.json` on startup (#3515)
This file is not actually used / useful - should metadata persistence
support be added in the future, it needs to be done with a new file such
that downgrades, that have the TODO logic unimplemented, don't break.
2022-03-18 12:36:50 +01:00
Jacek Sieka b3d80827fb
tns: checkpoint wal periodically while backfilling (#3516)
Witout this, we end up with a massive .wal file that needs to be
checkpointed on first startup (which takes a few minutes) - it's much
more efficient to do smaller checkpoints, it turns out.
2022-03-18 12:32:20 +01:00
Jacek Sieka 8395f7de8c
increase after-block attestation delay (#3518)
Recently, block processing times have been going up as the network grows
making early attestation riskier. Since blocks are big and attestations
are small (though numerous and therefore bandwidth-intense), it seems
better to wait a little bit longer after receiving a block, before we
publish the attestation.
2022-03-18 11:02:32 +00:00
Etan Kissling 12dc427535
introduce light client processor (#3509)
Adds `LightClientProcessor` as the pendant to `BlockProcessor` while
operating in light client mode. Note that a similar mechanism based on
async futures is used for interoperability with existing infrastructure,
despite light client object validation being done synchronously.
2022-03-17 23:26:56 +01:00
Etan Kissling 9f8894fb43
broadcast optimistic light client updates (#3499)
After proposing a new block, broadcasts a `OptimisticLightClientUpdate`.
Works for both locally proposed blocks as well as VC submitted ones.
2022-03-17 21:11:29 +01:00
Jacek Sieka 05ffe7b2bf
Prune `BlockRef` on finalization (#3513)
Up til now, the block dag has been using `BlockRef`, a structure adapted
for a full DAG, to represent all of chain history. This is a correct and
simple design, but does not exploit the linearity of the chain once
parts of it finalize.

By pruning the in-memory `BlockRef` structure at finalization, we save,
at the time of writing, a cool ~250mb (or 25%:ish) chunk of memory
landing us at a steady state of ~750mb normal memory usage for a
validating node.

Above all though, we prevent memory usage from growing proportionally
with the length of the chain, something that would not be sustainable
over time -  instead, the steady state memory usage is roughly
determined by the validator set size which grows much more slowly. With
these changes, the core should remain sustainable memory-wise post-merge
all the way to withdrawals (when the validator set is expected to grow).

In-memory indices are still used for the "hot" unfinalized portion of
the chain - this ensure that consensus performance remains unchanged.

What changes is that for historical access, we use a db-based linear
slot index which is cache-and-disk-friendly, keeping the cost for
accessing historical data at a similar level as before, achieving the
savings at no percievable cost to functionality or performance.

A nice collateral benefit is the almost-instant startup since we no
longer load any large indicies at dag init.

The cost of this functionality instead can be found in the complexity of
having to deal with two ways of traversing the chain - by `BlockRef` and
by slot.

* use `BlockId` instead of `BlockRef` where finalized / historical data
may be required
* simplify clearance pre-advancement
* remove dag.finalizedBlocks (~50:ish mb)
* remove `getBlockAtSlot` - use `getBlockIdAtSlot` instead
* `parent` and `atSlot` for `BlockId` now require a `ChainDAGRef`
instance, unlike `BlockRef` traversal
* prune `BlockRef` parents on finality (~200:ish mb)
* speed up ChainDAG init by not loading finalized history index
* mess up light client server error handling - this need revisiting :)
2022-03-17 17:42:56 +00:00
Etan Kissling 9a2b50d2c6
allow tagging light client specific libp2p messages (#3485)
The pre-release light client sync protocol defines additional Req/Resp
messages to be made available when `--serve-light-client-data` is set.
This patch extends the `{.libp2pProtocol.}` pragma with an optional
parameter to tag such light client sync protocol specific messages.
The corresponding protocols are only selectively registered with libp2p.
2022-03-17 16:09:18 +02:00
Jacek Sieka 8a63efc413
move `BlockId` to `spec` (#3511)
The spec implicitly talks about the slot of a block in several places,
and keeping it readily available is useful in a number of context -
might as well put this implicitly refereneced helper in the spec code
directly
2022-03-16 16:00:18 +01:00
Etan Kissling 88af3f2797
update to latest light client spec (#3508)
Adds the additional check to ensure `optimistic_header` is always after
`finalized_header` in `LightClientStore`, as introduced to the spec in
https://github.com/ethereum/consensus-specs/pull/2814
2022-03-16 12:56:38 +01:00
tersec 8fbcf29775
update unchanged specs/phase0/p2p-interface.md URL references from v1.1.9 to v1.1.10 (#3510) 2022-03-16 10:40:35 +00:00