Commit Graph

2836 Commits

Author SHA1 Message Date
Jacek Sieka 7ec1521c52
use unsigned literals (#3717)
in the hopes of avoiding potential for conversion bugs on i386
2022-06-08 11:09:33 +00:00
Tanguy 29297e9ce4
Bump libp2p (#3709) 2022-06-08 07:53:50 +02:00
Etan Kissling 72a46bd520
integrate light client into beacon node (#3557)
Adds a `LightClient` instance to the beacon node as preparation to
accelerate syncing in the future (optimistic sync).

- `--light-client-enable` turns on the feature
- `--light-client-trusted-block-root` configures block to start from

If no block root is configured, light client tracks DAG `finalizedHead`.
2022-06-07 19:01:11 +02:00
tersec 7078833f56
per-validator suggested fee recipients (#3652) 2022-06-06 16:58:47 +03:00
Jacek Sieka b35584632b
sync: remove `step` from sync client implementation (#3678)
* sync: remove `step` from sync client implementation

Deprecated in the spec:
https://github.com/ethereum/consensus-specs/pull/2856 - future PR:s will
deprecate server support as well.
2022-06-06 16:56:59 +03:00
tersec 1262e50fa5
document MEV/mergemock test script (#3694) 2022-06-06 16:55:02 +03:00
tersec 38737549ac
refactor fork consistency checking and gate compilation on it (#3704) 2022-06-04 19:15:15 +00:00
Dustin Brody 21200f4a64 fix false-positive in overlap between default {CAPELLA,SHARDING}_FORK_VERIONs 2022-06-04 14:52:03 +00:00
tersec faf4d4a001
initial Capella support in RuntimeConfig (#3698) 2022-06-03 14:42:40 +00:00
tersec ea113fc420
disallow non-(genesis, far-future) equal transition epochs (#3691) 2022-06-03 09:37:03 +00:00
tersec 7492f99f35
update CL spec URLs (#3696) 2022-06-03 09:01:58 +00:00
Eugene Kabanov 50f9596108
Eliminate rpc_types.nim usage. (#3692) 2022-06-02 09:39:08 +00:00
tersec ce143a1078
update CL spec URLs (#3690) 2022-06-01 15:52:45 +00:00
tersec 62bfe97bbe
fix ExecutionPayload(Header) JSON serialization (#3679) 2022-06-01 14:57:28 +02:00
Jacek Sieka 6a0433cb08
REST parameter defaults (#3689)
* add default BN for VC
* unify "default" beacon node across commands
* clean up validator exit docs
2022-06-01 10:47:52 +00:00
tersec f929980bf3
update 20 CL spec ref URLs (#3677) 2022-05-31 11:15:31 +00:00
tersec 48f9631ed1
remove JSON-RPC consensus layer API client (#3676)
* remove JSON-RPC consensus layer API client

* rm eth2_json_rpc_serialization
2022-05-31 11:05:15 +00:00
Etan Kissling 01efa93cf6
add light client (standalone) (#3653)
Introduces a new library for syncing using libp2p based light client
sync protocol, and adds a new `nimbus_light_client` executable that uses
this library for syncing. The new executable emits log messages when
new beacon block headers are received, and is integrated into testing.
2022-05-31 12:45:37 +02:00
Jacek Sieka f31f52e24a
fix missing frontfill index (fixes #3658) (#3675)
* fix key load duration log
* log broken frontfill block root
2022-05-31 10:09:01 +02:00
Dustin Brody f124f22f10
Version 22.5.2 2022-05-30 15:36:40 +00:00
Jacek Sieka 48f01186d6
fix unnecessary HashList/HashArray cache invalidation (#3660)
* SSZ `[]` -> `mitem`
* `[]` -> `item`

immutable access via mutable instance cannot rely on template
overloading, and `[]` cannot be a `func` because of special seq handling
in compiler.
2022-05-30 13:30:42 +00:00
tersec a3413963a1
update (or for one, remove) 15 CL spec ref URLs (#3671) 2022-05-30 12:24:43 +00:00
Dustin Brody 14dc3855f6
change proposer boost to 40% 2022-05-30 10:18:49 +00:00
tersec 01534b0431
🐼 (#3670)
* 🐼

* rm panda refs outside core module; preprocess text/ANSI artwork sources

* credit artwork to beatscribe
2022-05-30 08:25:27 +00:00
tersec bf1763fdef
consensus specs v1.2.0-rc.1 (#3661) 2022-05-25 13:49:29 +00:00
tersec dfd8cd22b7
bump nim-web3 and use engine API v1.0.0.alpha.9 (#3663) 2022-05-25 10:30:37 +00:00
Etan Kissling acf1ec43f8
serve light client data on `ropsten` by default (#3659)
Applies a `ropsten` testnet specific config default to serve
light client data on that network.
2022-05-24 13:04:11 +00:00
tersec b3d603f364
more CL spec URL updates to v1.2.0-rc.1 (#3657) 2022-05-24 08:26:35 +00:00
tersec d35d408fa7
remove deprecated JSON-RPC server (#3656)
* remove deprecated JSON-RPC server

* keep the command-line options around as no-ops, temporarily

* service -> server; JSON-RPC is still used elsewhere
2022-05-24 07:23:48 +00:00
Jacek Sieka 1101c745b9
document and clean up `ValidatorIndex` usage (#3651)
* document static vs dynamic range checking requirements
* add `vindices` iterator to iterate over valid validator indices in a
state
* clean up spec comments in general

* fixup

Co-authored-by: tersec <tersec@users.noreply.github.com>
2022-05-23 23:39:08 +00:00
tersec c73239f60b
CL spec URL updates to v1.2.0-rc.1 (#3655) 2022-05-23 19:30:24 +00:00
tersec 5773f71ac0
update engine API URLs to v1.0.0-alpha.9 (#3654) 2022-05-23 15:41:34 +00:00
Etan Kissling c808f17a37
update to latest light client libp2p protocol (#3623)
Incorporates the latest changes to the light client sync protocol based
on Devconnect AMS feedback. Note that this breaks compatibility with the
previous prototype, due to changes to data structures and endpoints.
See https://github.com/ethereum/consensus-specs/pull/2802
2022-05-23 14:02:54 +02:00
Zahary Karadjov f7eff8fcbe
Bump the version number 2022-05-21 13:11:11 +03:00
zah e7ce3cacd0
Add support for the Ropsten beacon chain (#3648) 2022-05-20 18:26:07 +03:00
zah 68fb3962c8
More spec-compliant handling of unknown fields in REST json (#3647)
* More spec-compliant handling of unknown fields in REST json

* Address review comments
2022-05-20 18:25:26 +03:00
tersec 50e84156cc
only send JWT headers in Bellatrix-capable networks (#3645) 2022-05-19 20:36:47 +00:00
Zahary Karadjov 592c246b26
Version 22.5.0 2022-05-17 23:09:30 +03:00
zah 18968e9dfa
Bugfix: Invalid blocks were produced in the presence of invalid deposits (#3639)
Since we were not verifying BLS signature in blocks that we produce,
we were failing to notice that some deposits need to be ignored (due
to having an invalid signature). Processing these deposits resulted
in a different ending state after the state transition which caused
our blocks to be rejected by the network.
2022-05-17 22:56:15 +03:00
zah 397033a7d6
Improve the compatibility with various Web3Signer versions and configurations (#3640)
* Some Web3Signer versions insist replying with text/plain messages
* When reading blocks, the Web3Signer uses upper-case fork identifiers
  instead of lower-case identifies like the Beacon API.
2022-05-17 19:55:03 +03:00
tersec 1177f33363
standardize on upcoming/specified engine API timeouts (#3637) 2022-05-17 13:57:33 +00:00
tersec e7ed7332b3
MEV REST serialization and call signatures (#3625) 2022-05-17 13:56:19 +00:00
zah a0a6dd2f63
Add a ncli tool for converting a regular keystore into a distributed one (#3634) 2022-05-17 16:50:49 +03:00
tersec 6e28ce425a
remove --use-jwt-debug (#3619) 2022-05-11 17:50:19 +00:00
Etan Kissling 8cfb630aa9
never request blocks before `safeSlot` in sync (#3512)
Follows up on https://github.com/status-im/nimbus-eth2/pull/3461 which
ensured that repeated `beaconBlocksByRange` requests get shrinked to
account for potential out-of-band advancements to `safeSlot`, with
similar logic for the initial request.
2022-05-10 13:46:14 +02:00
zah a2ba34f686
Implement all sync committee duties in the validator client (#3583)
Other changes:

* logtrace can now verify sync committee messages and contributions
* Many unnecessary use of pairs() have been removed for consistency
* Map 40x BN response codes to BeaconNodeStatus.Incompatible in the VC
2022-05-10 10:03:40 +00:00
zah 6d11ad6ce1
Support for distributed keystores with multiple remotes based on threshold signatures (#3616)
Other fixes:

* Fix bit rot in the `make prater-dev-deposit` target.
* Correct content-type in the responses of the Nimbus signing node
* Invalid JSON payload was being sent in the web3signer requests
2022-05-10 03:32:12 +03:00
Jacek Sieka 011e0ca02f
era file verification (#3605)
* era file verification

Implement and document era file verification

* era file states now come with block applied for easier verification
* clarify conflicting version handling
* document verification requirements
* remove count from name, use start-era, end-root to discover range

* remove obsolete todo

* abstract out block root loading
2022-05-10 03:28:46 +03:00
tersec fc75c3ce36
tracking update to MEV SSZ objects (#3622) 2022-05-09 14:55:15 +00:00
tersec 7ddba43ff9
Add signed MEV SSZ objects (#3621) 2022-05-09 11:19:20 +00:00
tersec eea33b82a9
add current MEV builder SSZ types (#3620) 2022-05-08 22:42:21 +00:00
tersec 104cc3053f
fcU on syncing newPayload syncing response (#3618) 2022-05-08 09:09:46 +02:00
Jacek Sieka c6915ed3e0
remove faststreams backend for eth2_network (#3617)
unmaintained, incompatible with altair+
2022-05-08 09:08:13 +02:00
Etan Kissling fec933432d
update presets / configs / REST config for v1.1.10 (#3614)
Updated outdated presets / configs / REST config to v1.1.10 specs.

- `TERMINAL_BLOCK_HASH_ACTIVATION_EPOCH` and `PROPOSER_SCORE_BOOST` are
  not yet used in `eth2-networks`, added configurability as TODOs.

- `MIN_ANCHOR_POW_BLOCK_DIFFICULTY` is no longer needed, put on ignore
  list as some Altair devnets still reference it.
2022-05-06 09:51:28 +02:00
Jacek Sieka 138c40161d
avoid unnecessary recompression in block protocol (#3598)
Blocks can be sent straight from compressed data sources

Co-authored-by: Etan Kissling <etan@status.im>
2022-05-05 11:00:02 +00:00
tersec 7bb40d28ae
ensure MAX_CHUNK_SIZE usage consistent in sync_protocol (#3615) 2022-05-05 09:17:14 +00:00
tersec 4a372410a4
use MAX_CHUNK_SIZE_BELLATRIX for signed Bellatrix blocks (#3613)
* use MAX_CHUNK_SIZE_BELLATRIX for signed Bellatrix blocks

* Update beacon_chain/networking/eth2_network.nim

Co-authored-by: Etan Kissling <etan@status.im>

* localPassC to localPassc

* check against maxChunkSize rather than constant

Co-authored-by: Etan Kissling <etan@status.im>
2022-05-05 05:45:35 +00:00
tersec 3088da1303
enable JWT by default (#3610) 2022-04-30 12:18:06 +00:00
tersec a1ae53fbdf
don't always broadcast sync contributions and aggregates on Altair topics (#3611) 2022-04-28 10:51:49 +00:00
zah abfef0a1b3
Fix invalid merkle proof generation when there is a large number of unprocessed deposits (#3607) 2022-04-27 12:39:41 +03:00
tersec 0d5bbbc405
mitigate Eth1Monitor being nil in merge scenario (#3600) 2022-04-26 12:25:19 +03:00
tersec 09b2f4c237
unhide --jwt-secret flag (#3604) 2022-04-19 11:34:55 +00:00
tersec 9e738a92b4
stylecheck fixes (#3595) 2022-04-15 12:46:56 +00:00
Jacek Sieka d0dbc4a8f9
Snappy revamp (#3564)
This PR makes the necessary adjustments to deal with the revamped snappy
API.

In practical terms for nimbus-eth2, there are performance increases to
gossip processing, database reading and writing as well as era file
processing. Exporting `.era` files for example, a snappy-heavy
operation, almost halves in total processing time:

Pre:

```
     Average,       StdDev,          Min,          Max,      Samples,         Test
      39.088,        8.735,       23.619,       53.301,           50, tState
     237.079,       46.692,      165.620,      355.481,           49, tBlocks
```

Post:

```
All time are ms
     Average,       StdDev,          Min,          Max,      Samples,         Test
      25.350,        5.303,       15.351,       41.856,           50, tState
     141.238,       24.164,       99.990,      199.329,           49, tBlocks
```
2022-04-15 09:44:06 +02:00
tersec ab1fac7236
post-merge Bellatrix block proposals (#3570)
* post-merge Bellatrix block proposals

* tolerate running without an Eth1Monitor better

* remove obsolete comment

* use correct empty receipts root

* handle invalid CLI parameters in parseCmdArg overloads
2022-04-14 20:15:34 +00:00
tersec 61ba308e13
stylecheck fixes (#3593) 2022-04-14 17:39:37 +02:00
Eugene Kabanov 5592c7c674
NoMonitor and removed clock check for SyncManager. (#3420)
* Add `NoMonitor` flag to stop SyncManager from monitoring sync situation.

* Remove `toleranceValue` and `PeerScoreHeadTooNew`.

Co-authored-by: Etan Kissling <etan@status.im>
2022-04-14 15:17:44 +02:00
tersec 28ba2d5544
stylecheck fixes (#3592) 2022-04-14 13:47:14 +03:00
tersec 06da9f4868
remove proposer boosting option (always enabled now) (#3586) 2022-04-13 12:14:00 +03:00
tersec 8b74847f85
only initialize JWT if useJwt is set (#3585) 2022-04-12 22:28:01 +00:00
Dustin Brody 6ab5cb1729
fix underlying issue 2022-04-12 20:17:32 +00:00
tersec ff6c581273
keep proposer boosting permanently enabled (#3565) 2022-04-12 12:06:30 +02:00
Zahary Karadjov 039bece917
version 22.4.0 2022-04-12 00:02:28 +03:00
Zahary Karadjov def69e2a06
Revert "More sparse state snapshots in the Gnosis network"
This reverts commit 557717b517.
2022-04-11 13:56:42 +03:00
Zahary Karadjov ac4e7723ea
Fix the build 2022-04-10 23:10:40 +03:00
Zahary Karadjov 557717b517
More sparse state snapshots in the Gnosis network 2022-04-09 18:07:36 +03:00
tersec 3796810bcd
rm unused imports (#3580) 2022-04-08 21:35:52 +00:00
Jacek Sieka d1a8daec7e
`gnosis-chain` -> `gnosis` (#3577)
* `gnosis-chain` -> `gnosis`

Use same name as LH/Teku throughout

* fixes #3504
* fixes large stack temporary that can cause crashes during genesis
detection
2022-04-08 23:11:37 +03:00
Zahary Karadjov d450681b15
Fix another off-by-one causing rejected sync contributions at period boundaries 2022-04-08 22:47:47 +03:00
Jacek Sieka f70ff38b53
enable `styleCheck:usages` (#3573)
Some upstream repos still need fixes, but this gets us close enough that
style hints can be enabled by default.

In general, "canonical" spellings are preferred even if they violate
nep-1 - this applies in particular to spec-related stuff like
`genesis_validators_root` which appears throughout the codebase.
2022-04-08 16:22:49 +00:00
Jacek Sieka f20c53be82
Revert "Polish, remove beta warning for validator monitor (#3531)" (#3578)
This reverts commit 540fc5a1af.
2022-04-08 15:05:17 +00:00
Jacek Sieka f9f4a49c65
stop calling `eth_protocolVersion` (fixes #3532)
no longer supported by geth
2022-04-08 13:08:35 +02:00
Jacek Sieka 7207185bc3
bump nim-ssz-serialization (#3576)
currently pointing to a branch that was merged and no longer exists
2022-04-07 20:08:20 +00:00
Jacek Sieka 540fc5a1af
Polish, remove beta warning for validator monitor (#3531)
Validator monitoring improves logging by giving more specific monitoring
information, and can now be seen as complete.

Previously, logging has focused on "Attestation sent" messages which
carry little informational value when things go wrong, and are overly
aggressive when everything works as expected (sending attestations is
the norm).

* lower "Attestation sent" log to `INFO`
* mark 1.7.0 as the start of the validator monitor feature - previous
versions had significant bugs in totals mode
2022-04-06 19:44:03 +03:00
Jacek Sieka 30eef0a369
Validator monitor polish (#3569)
* lower "Previous epoch attestation missing" to `NOTICE` for easier
filtering
* add delay logging to validator monitor logs
* simplify delay logging code post-`BeaconTime`
2022-04-06 09:23:01 +00:00
Jacek Sieka 6f4fa32c1d
remove bundled pyrmont support (#3568)
The `pyrmont` testnet has been discontinued.

For experiments, it's still possible to run pyrmont nodes by passing a
genesis/config, but this PR removes the bundled `--network:pyrmont`
option.

* update docs
* remove empty docs
* remove obsolete `eth2-stats` page
2022-04-05 12:59:25 +00:00
tersec 759a793764
use Eth1Monitor as abstraction; increase timeouts; handle newPayload 'accepted' (#3563) 2022-04-05 08:40:59 +00:00
Etan Kissling 509aa6c252
serve light client data on `prater` by default (#3559)
Applies a `prater` testnet specific config default to serve light client
data on that network.
2022-04-01 23:58:06 +02:00
tersec a18b39c9c1
JWT support (#3561) 2022-03-31 14:43:05 +00:00
zah f89c604fb0
Add Nimble lock file (#3560) 2022-03-30 22:00:03 +00:00
Jacek Sieka 5092fc41c7
use snappy-framed format for compressing bellatrix+ database entries (#3551)
`.era` files and Req/Resp protocols use framed formats - aligning the
database with these makes for less recompression work overall as gossip
is sent only once while req/resp repeats (potentially) - this also
allows efficient pruning-to-era where snappy-recompression is the major
cycle thief.
2022-03-29 11:33:06 +00:00
Etan Kissling 2c5f725543
reformat long lines (#3539)
Shortens some long lines by introducing temp variables and line breaks.
2022-03-29 09:15:42 +02:00
tersec d63b836f61
cap number of threads for --num-threads=0 (#3547)
* cap number of threads for --num-threads=0

* cap at 16 threads instead
2022-03-28 16:33:25 +00:00
tersec 9b43a76f2f
kiln beacon node (#3540)
* kiln bn

* use  version of beacon_chain_db

* have Eth1Monitor abstract more tightly over web3provider
2022-03-25 11:40:10 +00:00
tersec b37bf8c94b
remove TaintedString (#3546)
* remove TaintedString

* dumpDir is used as a helper function for more specific types of dumpDirs
2022-03-24 21:44:34 +00:00
Jacek Sieka e009728858
work around Nim assignment bug that breaks state pruning (#3545)
See https://github.com/nim-lang/Nim/issues/19613
2022-03-24 14:37:37 +00:00
Jacek Sieka 6983dacc26
fix bellatrix table names (#3544)
this should/will cause existing nimbus databases to revert to the altair
merge and resync with the new table name
2022-03-24 14:36:31 +01:00
Dustin Brody 3d9ef9b9b6
clean up action tracker 2022-03-24 10:57:20 +00:00
tersec 50f5754e3c
exists{Dir,File} -> {dir,file}Exists; rm unused imports (#3543) 2022-03-24 00:38:48 +00:00
Jacek Sieka bc4d4e1741
remove redundant `process_block` overloads (#3541) 2022-03-23 12:49:54 +00:00
Jacek Sieka bc80ac3be1
harden REST API `atSlot` against non-finalized blocks (#3538)
* harden validator API against pre-finalized slot requests
* check `syncHorizon` when responding to validator api requests too far
from `head`
* limit state-id based requests to one epoch ahead of `head`
* put historic data bounds on block/attestation/etc validator production API, preventing them from being used with already-finalized slots
* add validator block smoke tests
* make rest test create a new genesis with the tests running roughly in
the first epoch to allow testing a few more boundary conditions
2022-03-23 12:42:16 +01:00
Jacek Sieka 4207b127f9
era: load blocks and states (#3394)
* era: load blocks and states

Era files contain finalized history and can be thought of as an
alternative source for block and state data that allows clients to avoid
syncing this information from the P2P network - the P2P network is then
used to "top up" the client with the most recent data. They can be
freely shared in the community via whatever means (http, torrent, etc)
and serve as a permanent cold store of consensus data (and, after the
merge, execution data) for history buffs and bean counters alike.

This PR gently introduces support for loading blocks and states in two
cases: block requests from rest/p2p and frontfilling when doing
checkpoint sync.

The era files are used as a secondary source if the information is not
found in the database - compared to the database, there are a few key
differences:

* the database stores the block indexed by block root while the era file
indexes by slot - the former is used only in rest, while the latter is
used both by p2p and rest.
* when loading blocks from era files, the root is no longer trivially
available - if it is needed, it must either be computed (slow) or cached
(messy) - the good news is that for p2p requests, it is not needed
* in era files, "framed" snappy encoding is used while in the database
we store unframed snappy - for p2p2 requests, the latter requires
recompression while the former could avoid it
* front-filling is the process of using era files to replace backfilling
- in theory this front-filling could happen from any block and
front-fills with gaps could also be entertained, but our backfilling
algorithm cannot take advantage of this because there's no (simple) way
to tell it to "skip" a range.
* front-filling, as implemented, is a bit slow (10s to load mainnet): we
load the full BeaconState for every era to grab the roots of the blocks
- it would be better to partially load the state - as such, it would
also be good to be able to partially decompress snappy blobs
* lookups from REST via root are served by first looking up a block
summary in the database, then using the slot to load the block data from
the era file - however, there needs to be an option to create the
summary table from era files to fully support historical queries

To test this, `ncli_db` has an era file exporter: the files it creates
should be placed in an `era` folder next to `db` in the data directory.
What's interesting in particular about this setup is that `db` remains
as the source of truth for security purposes - it stores the latest
synced head root which in turn determines where a node "starts" its
consensus participation - the era directory however can be freely shared
between nodes / people without any (significant) security implications,
assuming the era files are consistent / not broken.

There's lots of future improvements to be had:

* we can drop the in-memory `BlockRef` index almost entirely - at this
point, resident memory usage of Nimbus should drop to a cool 500-600 mb
* we could serve era files via REST trivially: this would drop backfill
times to whatever time it takes to download the files - unlike the
current implementation that downloads block by block, downloading an era
at a time almost entirely cuts out request overhead
* we can "reasonably" recreate detailed state history from almost any
point in time, turning an O(slot) process into O(1) effectively - we'll
still need caches and indices to do this with sufficient efficiency for
the rest api, but at least it cuts the whole process down to minutes
instead of hours, for arbitrary points in time

* CI: ignore failures with Nim-1.6 (temporary)

* test fixes

Co-authored-by: Ștefan Talpalaru <stefantalpalaru@yahoo.com>
2022-03-23 09:58:17 +01:00
Etan Kissling 49673c4410
refer to `syncCommitteeMsgPool` consistently (#3537)
Updates a `sync_committee_msg_pool` reference to camelCase.
2022-03-23 07:46:48 +01:00
Etan Kissling b2b7b0bd56
serve libp2p protocol for light client sync (#3341)
This extends the `--serve-light-client-data` launch option to serve
locally collected light client data via libp2p.
Backfill of historic best `LightClientUpdate` is not yet implemented.
See https://github.com/ethereum/consensus-specs/pull/2802
2022-03-22 21:23:36 +01:00
Jacek Sieka 70270eeabe
better error messages on directory creation failure (#3536) 2022-03-22 17:06:21 +00:00
Jacek Sieka 4a237cb908
enable chronosStrictException (#3533)
* bump nim-json-rpc
2022-03-22 09:42:28 +01:00
Etan Kissling 8cc394ba49
extract DAG dependent init to own function (#3530)
During operation as a light client, the chain DAG is not available.
As a preparation, the beacon node initialization logic is divided into
parts depending on the presence of the chain DAG, and parts that are
always available (including a future light client mode).
This is a pure code move without semantic changes.
2022-03-21 17:52:15 +01:00
Jacek Sieka 361596c719
harden head update against missing parent (#3529)
in case BlockRef ends up in some lifetime leak

* fix duplicate head logging
2022-03-21 15:18:05 +01:00
Jacek Sieka 13fafe3a40
simplify unviable head pruning (#3528)
Also note bug that exists that potentially prevents states from being
pruned correctly
2022-03-21 09:20:26 +00:00
Etan Kissling fd1ffd62dd
update light client server for DAG failure modes (#3514)
Gracefully handles the new failure modes recently introduced to the DAG
as part of https://github.com/status-im/nimbus-eth2/pull/3513
Data that is deemed to exist but fails to load leads to an error log to
avoid suppressing logic errors accidentally. In `verifyFinalization`
mode, the assertions remain active.
2022-03-20 11:58:59 +01:00
Etan Kissling 04b851f775
fix light client data pruning (#3523)
When eliminating orphaned forks, light client data about blocks was also
deleted when the orphaned fork was referring to a state several slots
after the block. Linking light client data pruning with block deletion
instead of state deletion fixes this problem. Light client data always
refers to blocks and their immediate post-state.
2022-03-20 10:09:43 +01:00
Etan Kissling ca045900c8
extract chain DAG loading to separate function (#3527)
When transitioning from light client to full node the chain DAG will be
loaded separately from the rest of the beacon node initialization.
Extracting chain DAG loading to a separate function will allow reusing
a lot of the existing code. This code move doesn't change semantics.
2022-03-19 17:48:24 +01:00
Jacek Sieka ea1acd7397
fix loading when finalized checkpoint slot is missing block (#3525)
ref loop would stop one block early in this case - trying to load
everything in one loop ends up being pretty confusing..

* simplify finalizedBlocks topup by splitting it from the head loop /
query
2022-03-19 11:02:17 +00:00
Jacek Sieka e418497bb2
make attestation duty minimum offset relative to slot length (#3522) 2022-03-19 09:59:13 +01:00
Etan Kissling 637f1e2be6
simplify `computeEarliestLightClientSlot` (#3524)
Combine DAG and LC import tails in `computeEarliestLightClientSlot`.
2022-03-19 09:58:55 +01:00
Ștefan Talpalaru ea5c052016
enable multithreading by default (10-20% faster sync) (#3493) 2022-03-19 08:59:10 +01:00
Etan Kissling 18bd6df1b4
fix light client data collection for checkpoint sync (#3498)
When doing checkpoint sync, collecting light client data of known blocks
and states incorrectly assumes that `finalized_checkpoint` information
is also known. Hardens collection to only collect finalized checkpoint
data after `dag.computeEarliestLightClientSlot`.
2022-03-18 15:47:53 +01:00
Jacek Sieka d0223d1f28
fix finalized epoch ref loading on checkpoint start (#3517)
regression from #3513 that did not take tail into consideration when
loading epoch ancestor
2022-03-18 13:13:57 +01:00
tersec d11d61c745
engine API alpha.7 -> alpha.8 and a few remaining v1.1.9 to v1.1.0 CL spec URL updates (#3519) 2022-03-18 11:46:39 +00:00
Jacek Sieka 0db1e768e4
don't write `node-metadata.json` on startup (#3515)
This file is not actually used / useful - should metadata persistence
support be added in the future, it needs to be done with a new file such
that downgrades, that have the TODO logic unimplemented, don't break.
2022-03-18 12:36:50 +01:00
Jacek Sieka b3d80827fb
tns: checkpoint wal periodically while backfilling (#3516)
Witout this, we end up with a massive .wal file that needs to be
checkpointed on first startup (which takes a few minutes) - it's much
more efficient to do smaller checkpoints, it turns out.
2022-03-18 12:32:20 +01:00
Jacek Sieka 8395f7de8c
increase after-block attestation delay (#3518)
Recently, block processing times have been going up as the network grows
making early attestation riskier. Since blocks are big and attestations
are small (though numerous and therefore bandwidth-intense), it seems
better to wait a little bit longer after receiving a block, before we
publish the attestation.
2022-03-18 11:02:32 +00:00
Etan Kissling 12dc427535
introduce light client processor (#3509)
Adds `LightClientProcessor` as the pendant to `BlockProcessor` while
operating in light client mode. Note that a similar mechanism based on
async futures is used for interoperability with existing infrastructure,
despite light client object validation being done synchronously.
2022-03-17 23:26:56 +01:00
Etan Kissling 9f8894fb43
broadcast optimistic light client updates (#3499)
After proposing a new block, broadcasts a `OptimisticLightClientUpdate`.
Works for both locally proposed blocks as well as VC submitted ones.
2022-03-17 21:11:29 +01:00
Jacek Sieka 05ffe7b2bf
Prune `BlockRef` on finalization (#3513)
Up til now, the block dag has been using `BlockRef`, a structure adapted
for a full DAG, to represent all of chain history. This is a correct and
simple design, but does not exploit the linearity of the chain once
parts of it finalize.

By pruning the in-memory `BlockRef` structure at finalization, we save,
at the time of writing, a cool ~250mb (or 25%:ish) chunk of memory
landing us at a steady state of ~750mb normal memory usage for a
validating node.

Above all though, we prevent memory usage from growing proportionally
with the length of the chain, something that would not be sustainable
over time -  instead, the steady state memory usage is roughly
determined by the validator set size which grows much more slowly. With
these changes, the core should remain sustainable memory-wise post-merge
all the way to withdrawals (when the validator set is expected to grow).

In-memory indices are still used for the "hot" unfinalized portion of
the chain - this ensure that consensus performance remains unchanged.

What changes is that for historical access, we use a db-based linear
slot index which is cache-and-disk-friendly, keeping the cost for
accessing historical data at a similar level as before, achieving the
savings at no percievable cost to functionality or performance.

A nice collateral benefit is the almost-instant startup since we no
longer load any large indicies at dag init.

The cost of this functionality instead can be found in the complexity of
having to deal with two ways of traversing the chain - by `BlockRef` and
by slot.

* use `BlockId` instead of `BlockRef` where finalized / historical data
may be required
* simplify clearance pre-advancement
* remove dag.finalizedBlocks (~50:ish mb)
* remove `getBlockAtSlot` - use `getBlockIdAtSlot` instead
* `parent` and `atSlot` for `BlockId` now require a `ChainDAGRef`
instance, unlike `BlockRef` traversal
* prune `BlockRef` parents on finality (~200:ish mb)
* speed up ChainDAG init by not loading finalized history index
* mess up light client server error handling - this need revisiting :)
2022-03-17 17:42:56 +00:00
Etan Kissling 9a2b50d2c6
allow tagging light client specific libp2p messages (#3485)
The pre-release light client sync protocol defines additional Req/Resp
messages to be made available when `--serve-light-client-data` is set.
This patch extends the `{.libp2pProtocol.}` pragma with an optional
parameter to tag such light client sync protocol specific messages.
The corresponding protocols are only selectively registered with libp2p.
2022-03-17 16:09:18 +02:00
Jacek Sieka 8a63efc413
move `BlockId` to `spec` (#3511)
The spec implicitly talks about the slot of a block in several places,
and keeping it readily available is useful in a number of context -
might as well put this implicitly refereneced helper in the spec code
directly
2022-03-16 16:00:18 +01:00
Etan Kissling 88af3f2797
update to latest light client spec (#3508)
Adds the additional check to ensure `optimistic_header` is always after
`finalized_header` in `LightClientStore`, as introduced to the spec in
https://github.com/ethereum/consensus-specs/pull/2814
2022-03-16 12:56:38 +01:00
tersec 8fbcf29775
update unchanged specs/phase0/p2p-interface.md URL references from v1.1.9 to v1.1.10 (#3510) 2022-03-16 10:40:35 +00:00
Jacek Sieka c64bf045f3
remove StateData (#3507)
One more step on the journey to reduce `BlockRef` usage across the
codebase - this one gets rid of `StateData` whose job was to keep track
of which block was last assigned to a state - these duties have now been
taken over by `latest_block_root`, a fairly recent addition that
computes this block root from state data (at a small cost that should be
insignificant)

99% mechanical change.
2022-03-16 08:20:40 +01:00
Etan Kissling 6d1d31dd01
avoid re-requesting finalized blocks during sync (#3461)
When a `beaconBlocksByRange` response advances the `safeSlot`, but later
has errors, the sync queue keeps repeating that same request until it is
fulfilled without errors. Data up through `safeSlot` is considered to be
immutable, i.e., finalized, so re-requesting that data is not useful.
By advancing the sync progress in that scenario, those redundant query
portions can be avoided. Note, the finalized block _itself_ is always
requested, even in the initial request. This behaviour is kept same.
2022-03-15 18:56:56 +01:00
Jacek Sieka a3bd01b58d
move dependent root computations to `BeaconState` / `EpochRef` (#3478)
* fewer deps on `BlockRef` traversal in anticipation of pruning
* allows identifying EpochRef:s by their shuffling as a first step of
* tighten error handling around missing blocks

using the zero hash for signalling "missing block" is fragile and easy
to miss - with checkpoint sync now, and pruning in the future, missing
blocks become "normal".
2022-03-15 09:24:55 +01:00
Etan Kissling a08114e996
libp2p light client gossip validation (#3486)
When `--serve-light-client-data` is specified, provides stability on the
`optimistic_light_client_update` GossipSub topic.
2022-03-14 14:05:38 +01:00
tersec f550eb2f17
fix two typos (#3491) 2022-03-14 12:50:23 +00:00
Etan Kissling 29e5a4a752
error and progress codes for light client sync (#3490)
When syncing as a light client, different behaviour is needed to handle
the various ways how errors may occur. The existing logic for blocks can
also be applied to light client objects:
- `Invalid`: Malformed object that is clearly an error by its producer.
- `MissingParent`: More data is needed to decide applicability.
- `UnviableFork`: Object may be valid but will never apply on this fork.
- `Duplicate`: No errors were encountered but the object was not useful.
2022-03-14 10:25:54 +01:00
Ștefan Talpalaru 276762958e
Windows: disable status bar (#3484)
It can randomly lock inside Windows terminal emulators. Better play it
safe.
2022-03-14 10:19:50 +01:00
Mamy Ratsimbazafy 9fd7305e26
Cleanup RPC pubkey handling (#3489) 2022-03-13 08:12:45 +01:00
Etan Kissling ae408c279a
add option to collect light client data (#3474)
Light clients require full nodes to serve additional data so that they
can stay in sync with the network. This patch adds a new launch option
`--import-light-client-data` to configure what data to make available.
For now, data is only kept in memory; it is not persisted at this time.
Note that data is only locally collected, a separate patch is needed to
actually make it availble over the network. `--serve-light-client-data`
will be used for serving data, but is not functional yet outside tests.
2022-03-11 21:28:10 +01:00
tersec 21b71bd29c
update URL and document Nim bug blocking further genericizing cleanups (#3483) 2022-03-11 15:03:47 +00:00
Jacek Sieka d0183ccd77
Historical state reindex for trusted node sync (#3452)
When performing trusted node sync, historical access is limited to
states after the checkpoint.

Reindexing restores full historical access by replaying historical
blocks against the state and storing snapshots in the database.

The process can be initiated or resumed at any point in time.
2022-03-11 12:49:47 +00:00
Jacek Sieka 4363215a32
relax `BlockRef` database assumptions (#3472)
* remove `getForkedBlock(BlockRef)` which assumes block data exists but
doesn't support archive/backfilled blocks
* fix REST `/eth/v1/beacon/headers` request not returning
archive/backfilled blocks
* avoid re-encoding in REST block SSZ requests (using `getBlockSSZ`)
2022-03-11 13:08:17 +01:00
Etan Kissling 438aa17f7b
allow `SyncCommitteePeriod` as libp2p params (#3481)
Adds a converter to allow using `SyncCommitteePeriod` as parameter for
libp2p messages (used in pre-release light client sync protocol).
2022-03-11 13:04:08 +01:00
Tanguy f589bf2119
Peer dialing/kicking system overhaul (#3346)
* Force dial + excess peer trimmer
* Ensure we always have outgoing peers
* Add configurable hard-max-peers
2022-03-11 10:51:53 +00:00
Zahary Karadjov 023fa3b562
Merge branch 'stable' into unstable 2022-03-10 16:49:18 +02:00
Zahary Karadjov 13b264d509
Final v22.3.0: Add examples for migrating from JSON-RCP to REST 2022-03-10 16:46:11 +02:00
Zahary Karadjov 7b6e36ba6d
v22.3.1 2022-03-09 19:20:01 +02:00
Zahary Karadjov aa3bdb1228
Helpful error message when the user fails to use an array type in TOML
This applies to fields such as `web3-url` which are mapped to array
in TOML in a way that may surprise the user.
2022-03-09 19:11:41 +02:00
Etan Kissling 64242d9c84
add support for `ResourceUnavailable` p2p error (#3476)
The `p2p-interface.md` spec defines a `ResourceUnavailable` error to
return in situations where data that exists on the network is locally
unavailable, e.g., when a block within `MIN_EPOCHS_FOR_BLOCK_REQUESTS`
is requested by `BeaconBlocksByRange` but cannot be provided. This patch
adds support for that additional error code.
2022-03-09 14:03:58 +00:00
Tanguy 266fd98a13
Don't store invalid gossipsub messages (#3471) 2022-03-09 11:30:31 +01:00
Etan Kissling 41c820bc66
`shortLog` for light client types (#3473)
Adds log formatters for light client types.
2022-03-09 11:30:15 +01:00
Etan Kissling 5a3ba5d968
update to pre-release light client sync protocol (#3465)
This adopts the spec sections of the pre-release proposal of the libp2p
based light client sync protocol, and also adds a test runner for the
new accompanying tests. While the release version of the light client
sync protocol contains conflicting definitions, it is currently unused,
and the code specific to the pre-release proposal is marked as such.
See https://github.com/ethereum/consensus-specs/pull/2802
2022-03-08 13:21:56 +01:00
Etan Kissling aaa5a5ad40
add `start_slot` overload for sync periods (#3469)
Adds a `start_slot` overload for `SyncCommitteePeriod` as a shortcut for
`period.start_epoch.start_slot`.
2022-03-08 11:38:58 +01:00
Zahary Karadjov 7340e7cab9
v22.3.0
* Deprecates the JSON-RPC API
2022-03-07 21:49:12 +02:00
Zahary Karadjov 542e645bed
Fix off-by-one error in determining the sync committee when POSTing sync committee messages through the REST API 2022-03-07 21:49:12 +02:00
Zahary Karadjov 5ef2ce4069
Fix #3463 (validator index-out-of-bound errors triggered through the REST API) 2022-03-07 21:49:12 +02:00
Etan Kissling 8955edf158
allow using `BlockId` as key in tables (#3467)
`BlockId` is a type that bundles a block root with its slot number.
The type can be useful as key in tables that deal with non-finalized
blocks (not uniquely identified by slot) and also support pruning
(drop data about older blocks by slot). Instead of creating a custom
type for those use cases, this patch suggests implementing `hash` for
`BlockId` to re-use the existing type.
2022-03-07 14:56:58 +01:00
Etan Kissling 7d7bfa1299
add `toBeaconBlockHeader` overload (#3468)
Overloads `toBeaconBlockHeader` for `ForkedTrustedSignedBeaconBlock`.
2022-03-07 14:56:43 +01:00
Zahary Karadjov e6723ddb24
Allow running Nimbus as a Windows service (--run-as-service) 2022-03-05 15:53:47 +02:00
zah cdeae90806
Add support for TOML config files (--config-file) (#3442) 2022-03-05 04:33:15 +02:00
Ștefan Talpalaru e4b7dbf330
--stop-at-synced-epoch (#3464)
* --stop-at-synced-epoch

This allows benchmarking the initial sync (only forward sync, 1s error
margin). Might be useful in CI, with a timeout, as a sanity check.
2022-03-04 18:38:01 +01:00
Etan Kissling a84ab5d47f
validate `fork_version` as light client (#3459)
The spec does not provide code for validating the `fork_version` field
of `LightClientUpdate`. However, we can use our own logic for additional
validation of that field. The spec's python test suite sets up states
that do not follow the fork schedule (e.g., that use Altair fork version
before Altair fork epoch), which complicates upstreaming this as code.
2022-03-04 17:09:33 +01:00
Mamy Ratsimbazafy ef7e8bdbd2
Minify slashing protection before SQLite (#3393) 2022-03-04 16:43:34 +02:00
zah 8967f9cf01
Work-around the sizeof change in behavior introduced in Nim 1.6 (#3462) 2022-03-04 10:52:49 +02:00
tersec c18cd8ee0c
rename random -> prev_randao in Bellatrix for CL specs v1.1.10 (#3460) 2022-03-03 16:08:14 +00:00
Etan Kissling 47d7814518
update light client to v1.1.10 spec (#3457)
Adopts the changes introduced in the v1.1.10 ETH consensus-specs:
- Introduces `is_finality_update` helper
- Ensures `optimistic_header` always >= `finalized_header`
- Updates spec references
2022-03-03 14:03:08 +01:00
Zahary Karadjov 4c01b77773
The remote Keymanager API was not using the URLs indicated in the spec 2022-03-03 11:10:00 +02:00
Etan Kissling 3ffab01b07
Refactor and optimize sync logs. (#3451)
* Refactor and optimize logs.

* Introduce shortLog(SyncRequest).

* Address review comment.

* make sync queue logs more consistent

Adds a few minor logging improvements:
- Fixes a typo (`was happened` -> `has happened`)
- Avoids passing `reset_slot` argument to log statement multiple times
- Uses same `rewind_to_slot` label when logging in both sync directions
- Consistent rewind point logging

Co-authored-by: cheatfate <eugene.kabanov@status.im>
2022-03-03 09:05:33 +01:00
Etan Kissling 33d084192f
consistent style in `light_client_sync.nim` (#3450)
Uses consistent formatting in `light_client_sync.nim`, always refers to
fork-dependent light client objects in full qualified notation, moves
`get_safety_threshold` helper function to same location as in the spec.
2022-03-02 11:44:42 +01:00
tersec f0ada15dac
automated CL spec ref URL updates from v1.1.9 to v1.1.10 (#3455) 2022-03-02 10:00:21 +00:00
tersec 7b3d9d4e14
use v1.1.10 CL spec test vectors (#3454) 2022-03-02 07:26:17 +00:00
Jacek Sieka 12ed537f75
catch wrong-fork-blocks earlier (#3444)
Can't apply a phase0 block to a later phase state and vice versa.

Since instantiation has been a topic, pre/post c file size:

```
424K	@mspec@sstate_transition.nim.c
892K	@mspec@sstate_transition_block.nim.c
```

```
288K	@mspec@sstate_transition.nim.c
880K	@mspec@sstate_transition_block.nim.c
```
2022-02-28 12:58:34 +00:00
Etan Kissling 961c02fcba
document `GeneralizedIndex` constants (#3443)
Updates the spec references for `GeneralizedIndex` constants used by the
light client sync protocol, and adds a short explanation how they are
derived and which SSZ fields they refer to.
2022-02-28 13:34:57 +01:00
tersec ef9767eb7a
implement --jwt-secret and HS256 JWT/JWS signing for engine API alpha.7 (#3440) 2022-02-27 16:55:02 +00:00
Jacek Sieka 40a4c01086
chaindag: don't keep backfill block table in memory (#3429)
This PR names and documents the concept of the archive: a range of slots
for which we have degraded functionality in terms of historical access -
in particular:

* we don't support rewinding to states in this range
* we don't keep an in-memory representation of the block dag

The archive de-facto exists in a trusted-node-synced node, but this PR
gives it a name and drops the in-memory digest index.

In order to satisfy `GetBlocksByRange` requests, we ensure that we have
blocks for the entire archive period via backfill. Future versions may
relax this further, adding a "pre-archive" period that is fully pruned.

During by-slot searches in the archive (both for libp2p and rest
requests), an extra database lookup is used to covert the given `slot`
to a `root` - future versions will avoid this using era files which
natively are indexed by `slot`. That said, the lookup is quite
fast compared to the actual block loading given how trivial the table
is - it's hard to measure, even.

A collateral benefit of this PR is that checkpoint-synced nodes will see
100-200MB memory usage savings, thanks to the dropped in-memory cache -
future pruning work will bring this benefit to full nodes as well.

* document chaindag storage architecture and assumptions
* look up parent using block id instead of full block in clearance
(future-proofing the code against a future in which blocks come from era
files)
* simplify finalized block init, always writing the backfill portion to
db at startup (to ensure lookups work as expected)
* preallocate some extra memory for finalized blocks, to avoid immediate
realloc
2022-02-26 19:16:19 +01:00
Jacek Sieka 92e7e288e7
Ignore seen aggregates (#3439)
https://github.com/ethereum/consensus-specs/pull/2225 removed an ignore
rule that would filter out duplicate aggregates from gossip publishing -
however, this causes increased bandwidth and CPU usage as discussed in
https://github.com/ethereum/consensus-specs/issues/2183 - the intent is
to revert the removal and reinstate the rule.

This PR implements ignore filtering which cuts down on CPU usage (fewer
aggregates to validate) and bandwidth usage (less fanout of duplicates)
- as #2225 points out, this may lead to a small increase in IHAVE
messages.
2022-02-25 17:15:39 +01:00
Tanguy 1bfbcc48b6
Bump libp2p (#3438) 2022-02-25 13:22:48 +01:00
zah c29aa9d846
Support for Gnosis Chain (#3415)
* Support for Gnosis Chain

`make gnosis-chain-build` will build the Nimbus gnosis chain binary,
stored in `build/nimbus_beacon_node_for_gnosis_chain`.

`make gnosis-chain` will connect to the network.

Other changes:

* Restore compilation with -d:has_genesis_detection
* Removed Makefile target related to testnet0 and testnet1
* Added more debug logging for failed peer handshakes
* Report misconfigured builds which try to embed network metadata
that is incompatible with the currently selected const preset.
* Don't bundle network metadata in minimal builds, as they are not compatible
2022-02-25 10:22:44 +02:00
Ștefan Talpalaru ebba093362
Nim-1.6 compatibility (#3434) 2022-02-25 10:19:12 +02:00
tersec fef71a78a0
bump nim-web3 for random -> prevRandao rename (#3435) 2022-02-24 18:01:48 +01:00
zah 9c1ff78f84
Fix a reward calculation bug affecting Prater epoch 64781 (#3428)
To calculate the deltas correctly, the `process_inactivity_updates` function
must be called before the rewards and penalties processing code in order to
update the `inactivity_scores` field in the state. This would have required
duplicating more logic from the spec in the ncli modules, so I've decided to
pay the price of introducing a run-time copy of the state at each epoch which
eliminates the need to duplicate logic (both for this fix and the previous one).

Other changes:

* Fixes for the read-only mode of the `BeaconChainDb`
* Fix an uint64 underflow in the debug output procedure for printing
  balance deltas
* Allow Bellatrix states in the reward computation helpers
2022-02-22 14:14:17 +02:00
tersec 7de3f00f35
generic putCorruptState; {Merge=>Bellatrix}BeaconStateNoImmutableValidators (#3427) 2022-02-21 12:55:56 +01:00
Jacek Sieka adfe655b16
db: make block loading generic (#3413)
Streamline lookup with Forky and BeaconBlockFork (then we can do the
same for era)

We use type to avoid conditionals, as fork is often already known at a
"higher" level.

* load blockid before loading block by root - this is needed to map root
to slot and will eventually be done via block summary table for "old"
blocks

Co-authored-by: tersec <tersec@users.noreply.github.com>
2022-02-21 09:48:02 +01:00
tersec 84588b34da
var => let in specs/ and tests/ (#3425) 2022-02-20 20:13:06 +00:00
Etan Kissling 9790c4958b
converter function for reducing blocks to headers (#3410)
This introduces a function to convert `SignedBeaconBlock` to just their
`BeaconBlockHeader` and updates the usages for reduced code duplication.
2022-02-18 21:35:52 +01:00
Jacek Sieka a88427bd39
ncli_db: more readonly support (#3411)
Update several `ncli_db` commands to run in readOnly mode, allowing them
to be used with a running instance - in particular era export.

* export all eras by default
* skip already-exported eras
2022-02-18 07:37:44 +01:00
tersec 79761c78a4
proc -> func, mainly in spec/state transition and adjecent modules (#3405) 2022-02-17 11:53:55 +00:00
Jacek Sieka 87e98b9e54
Revert "bump submodules (#3366)" (#3406)
This reverts commit 6e1ad080e8.
2022-02-17 12:50:37 +01:00
tersec 5eecb9a21f
rename no{R=>r}eturn, no{I=>i}init, short{l=>L}og, E{T=>t}h2Node, Beacon{c=>C}hainDB (#3403) 2022-02-16 23:24:44 +01:00
Jacek Sieka 7db5647a6e
clean up / document init (#3387)
* clean up / document init

* drop `immutable_validators` data (pre-altair)
* document versions where data is first added
* avoid needlessly loading genesis block data on startup
* add a few more internal database consistency checks
* remove duplicate state root lookup on state load

* comment
2022-02-16 16:44:04 +01:00
Ștefan Talpalaru 6e1ad080e8
bump submodules (#3366)
and add Nim-1.6 compatibility
2022-02-16 13:41:50 +02:00
Eugene Kabanov 3a80b9951c
VC: Fix forks handling. (#3389)
* Trying to debug the finalization issue.

* Add debug logs to understand signature issue.

* Remove all the debugging helpers.

* Initial commit.

* Address review comments.

* Remove unneeded checks for empty fork schedule.

* Fix bellatrix ExecutionAddress serialization/deserialization procedures.
2022-02-16 12:31:23 +01:00
tersec 254e0fe2e2
avoid nimZeroMem and stack usage in is_merge_transition_complete and is_merge_transition_block (#3399) 2022-02-16 07:16:01 +00:00
Dustin Brody e1dbcfc02e
add --safe-slots-to-import-optimistically option 2022-02-15 23:08:49 +02:00
Zahary Karadjov 8e0330050d
Version 1.7.0 2022-02-15 22:55:29 +02:00
Zahary Karadjov c672628be8 Hotfix: Fix a race condition leading to a busy loop preventing progress in Eth1 syncing 2022-02-15 22:45:55 +02:00
Ștefan Talpalaru 496d0266ec
bump nim-metrics (#3392) 2022-02-14 21:57:06 +01:00
tersec 2275fad335
only show setting up doppelganger detection log message if enabled (#3391)
* only show setting up doppelganger detection log message if enabled

* correct indentation
2022-02-14 19:24:38 +00:00
Etan Kissling 6849536742 fix `firstSlot` computation for backfill sync
When initializing backfill sync, the implementation intends to start at
the first unknown slot (`1` before tail). However, an incorrect variable
is passed, and backfill sync actually starts at the tail slot instead.
This patch corrects this by passing the intended variable. The problem
was introduced with the original backfill implementation at #3263.
2022-02-14 18:53:38 +02:00
Zahary Karadjov 922a0d264c Add CORS support for the REST services
The added options work in opt-in fashion. If they are not specified,
the server will respond to all requests as if the CORS specification
doesn't exist. This will result in errors in CORS-enabled clients.

Please note that future versions may support more than one allowed
origin. The option names will stay the same, but the user will be
able to repeat them on the command line (similar to other options
such as --web3-url).

To be documented in the guide in a separate PR.
2022-02-14 18:52:17 +02:00
Etan Kissling d1f97e209a
remove unused `sleepTime` from `SyncManager` (#3384)
The `SyncManager` has a leftover optional `sleepTime` parameter in
its constructor that used to configure the sync loop polling rate.
This parameter was replaced with a constant in #1602 and is no longer
functional. This patch removes the `sleepTime` leftovers.
2022-02-14 12:05:01 +01:00
Etan Kissling a28900c348
fix slot number display during sync (#3383)
#3304 introduced a regression to the sync status string displayed in the
status bar; during the main forward sync, the current slot is no longer
reported and always displays as `0`. This patch corrects the computation
to accurately report the current slot once more.
2022-02-14 12:04:04 +01:00
tersec 873a8ec1e6
use isZeroMemory for Eth2Digest comparisons (#3386)
* use isZeroMemory for Eth2Digest comparisons

* use Eth2Digest.isZero abstraction
2022-02-14 05:26:19 +00:00
Eugene Kabanov 1a0bcf0b02
Fix #3267 (#3367)
* Initial commit.

* One more fix.

* Trying to debug the finalization issue.

* Add debug logs to understand signature issue.

* Restore hash_tree_root calculation.

* Remove all the debugging helpers.

* Add `slot` check.

* Address review comment.
2022-02-13 16:21:55 +01:00
Etan Kissling 15fc7534cf
remove unused `maxStatusAge` from `SyncManager` (#3382)
The `SyncManager` has a leftover optional `maxStatusAge` parameter in
its constructor that used to configure the libp2p `Status` polling rate.
This parameter was replaced with a constant in #1827 and is no longer
functional. This patch removes the `maxStatusAge` leftovers.
2022-02-13 16:17:13 +01:00
Jacek Sieka 1f89b7f7b9
speed up trusted node backfill (#3371)
With these changes, we can backfill about 400-500 slots/sec, which means
a full backfill of mainnet takes about 2-3h.

However, the CPU is not saturated - neither in server nor in client
meaning that somewhere, there's an artificial inefficiency in the
communication - 16 parallel downloads *should* saturate the CPU.

One plasible cause would be "too many async event loop iterations" per
block request, which would introduce multiple "sleep-like" delays along
the way.

I can push the speed up to 800 slots/sec by increasing parallel
downloads even further, but going after the root cause of the slowness
would be better.

* avoid some unnecessary block copies
* double parallel requests
2022-02-12 12:09:59 +01:00
Jacek Sieka 40fe8f5336 fix missing backfill when restarting node
When node is restarted before backfill has started but after some blocks
have finalized with forward sync, we would not start the backfill.

* also clean up one last `SomeSome`
2022-02-11 23:08:50 +02:00
Jacek Sieka 1760f4d7a7
move wallet/deposit commands to separate files (#3372)
These commands have little to do with the "normal" beacon node operation
- ergo, they deserve to live in their own module.

* clean up imports/exports
2022-02-11 21:40:49 +01:00
Eugene Kabanov b4eb150b9a
Revert restAccept workaround. (#3369)
Bump fixed version of nim-presto.
2022-02-11 12:01:45 +01:00
Ștefan Talpalaru 70b38e37e6
Nim GC metrics for the main thread (#3108)
* Nim GC metrics for the main thread
2022-02-08 20:19:21 +01:00
Eugene Kabanov 40c77e5928
Remote KeyManager API and number of fixes/tests for KeyManager API (#3360)
* Initial commit.

* Fix current test suite.

* Fix keymanager api test.

* Fix wss_sim.

* Add more keystore_management tests.

* Recover deleted isEmptyDir().

* Add `HttpHostUri` distinct type.
Move keymanager calls away from rest_beacon_calls to rest_keymanager_calls.
Add REST serialization of RemoteKeystore and Keystore object.
Add tests for Remote Keystore management API.
Add tests for Keystore management API (Add keystore).
Fix serialzation issues.

* Fix test to use HttpHostUri instead of Uri.

* Add links to specification in comments.

* Remove debugging echoes.
2022-02-07 22:36:09 +02:00
Jacek Sieka c7abc97545
harden and speed up block sync (#3358)
* harden and speed up block sync

The `GetBlockBy*` server implementation currently reads SSZ bytes from
database, deserializes them into a Nim object then serializes them right
back to SSZ - here, we eliminate the deser/ser steps and send the bytes
straight to the network. Unfortunately, the snappy recoding must still
be done because of differences in framing.

Also, the quota system makes one giant request for quota right before
sending all blocks - this means that a 1024 block request will be
"paused" for a long time, then all blocks will be sent at once causing a
spike in database reads which potentially will see the reading client
time out before any block is sent.

Finally, on the reading side we make several copies of blocks as they
travel through various queues - this was not noticeable before but
becomes a problem in two cases: bellatrix blocks are up to 10mb (instead
of .. 30-40kb) and when backfilling, we process a lot more of them a lot
faster.

* fix status comparisons for nodes syncing from genesis (#3327 was a bit
too hard)
* don't hit database at all for post-altair slots in GetBlock v1
requests
2022-02-07 19:20:10 +02:00
tersec bf3ef987e4
deactivate doppelganger protection during genesis (#3362)
* deactivate Doppelganger Protection during genesis

* also don't actually flag supposed-doppelgangers (because they're before broadcastStartEpoch) on GENESIS_SLOT start
2022-02-07 07:12:36 +02:00
Jacek Sieka 6f10e651ff
rest: fix ssz preference string (#3357) 2022-02-04 15:26:27 +02:00
tersec e0fb5d95a6
remove --subscribe-all{att,sync}nets (#3359) 2022-02-04 12:34:03 +00:00
tersec 02349b4181
update to engine API alpha.6 (#3351) 2022-02-04 12:12:19 +00:00
tersec d358299875
fork choice proposer boosting support (#3349)
* fork choice proposer boosting support

* detect nodeDelta underflow/overflow
2022-02-04 12:59:40 +01:00
Jacek Sieka a50e21e229
fix doppelganger detection logging
* update action tracker on dependent-root-changing reorg (instead of
epoch change)
* don't try to log duties while syncing - we're not tracking actions yet
* fix slot used for doppelganger loss detection
2022-02-04 12:25:32 +01:00
Jacek Sieka 49282e9477
val_mon: register locally produced aggregates (#3352)
These use a separate flow, and were previously only registered from the
network

* don't log successes in totals mode (TMI)
* remove `attestation-sent` event which is unused
2022-02-04 08:33:20 +01:00
Zahary Karadjov 215caa21ae Eth1 monitor fixes
* Fix a resource leak introduced in https://github.com/status-im/nimbus-eth2/pull/3279

* Don't restart the Eth1 syncing proggress from scratch in case of
  monitor failures during Eth2 syncing.

* Switch to the primary operator as soon as it is back online.

* Log the web3 credentials in fewer places

Other changes:

The 'web3 test' command has been enhanced to obtain and print more
data regarding the selected provider.
2022-02-03 14:01:55 +02:00
tersec 8e6a920bf4
rename MERGE_FORK_EPOCH to BELLATRIX_FORK_EPOCH (#3350)
* rename MERGE_FORK_EPOCH to BELLATRIX_FORK_EPOCH

* fix REST test rules
2022-02-02 14:06:55 +01:00
Jacek Sieka ff4f2a6b6c
better log on finalized slot failure 2022-02-01 21:23:18 +01:00
Tanguy bcd7b4598c
Tune peering (#3348)
- Request metadata_v2 (altair) by default instead of the v1
- Change the metadata pinger to a 3 failure-then-kick, instead of being time based
- Update kicker scorer to take into account topics which we're not subscribed to, to be sure that we will be able to publish correctly
- Add some metrics to give "fanout" health (in the same spirit of mesh health)
2022-02-01 18:20:55 +01:00
tersec 0c814f49ee
rename sync_{committee_,}aggregate and execute_payload -> notify_new_payload (#3347) 2022-02-01 07:31:53 +00:00
EmilIvanichkovv 336403d18b Refactor `handleValidatorExitCommand`
Make `validator exit command` work both with `JSON-RPC` and `REST` APIs
Fix problem with specifying rest-url using `localhost`
Change back exit error messages in `state_transition_block`
2022-02-01 01:24:05 +02:00
Jacek Sieka 3df9ffca9f val-mon: remove redundant `_total` suffix from counters
It turns out nim-metrics adds this suffix on its own - it also turns out
some of the names are non-conventional and need follow-up.
2022-01-31 18:51:24 +02:00
tersec c9aa1bee01
spec URL updates (#3342) 2022-01-31 09:56:59 +00:00
Jacek Sieka ad327a8769
Fix counters in validator monitor totals mode (#3332)
The current counters set gauges etc to the value of the _last_ validator
to be processed - as the name of the feature implies, we should be using
sums instead.

* fix missing beacon state metrics on startup, pre-first-head-selection
* fix epoch metrics not being updated on cross-epoch reorg
2022-01-31 08:36:29 +01:00
Jacek Sieka d583e8e4ac
Store finalized block roots in database (3s startup) (#3320)
* Store finalized block roots in database (3s startup)

When the chain has finalized a checkpoint, the history from that point
onwards becomes linear - this is exploited in `.era` files to allow
constant-time by-slot lookups.

In the database, we can do the same by storing finalized block roots in
a simple sparse table indexed by slot, bringing the two representations
closer to each other in terms of conceptual layout and performance.

Doing so has a number of interesting effects:

* mainnet startup time is improved 3-5x (3s on my laptop)
* the _first_ startup might take slightly longer as the new index is
being built - ~10s on the same laptop
* we no longer rely on the beacon block summaries to load the full dag -
this is a lot faster because we no longer have to look up each block by
parent root
* a collateral benefit is that we no longer need to load the full
summaries table into memory - we get the RSS benefits of #3164 without
the CPU hit.

Other random stuff:

* simplify forky block generics
* fix withManyWrites multiple evaluation
* fix validator key cache not being updated properly in chaindag
read-only mode
* drop pre-altair summaries from `kvstore`
* recreate missing summaries from altair+ blocks as well (in case
database has lost some to an involuntary restart)
* print database startup timings in chaindag load log
* avoid allocating superfluos state at startup
* use a recursive sql query to load the summaries of the unfinalized
blocks
2022-01-30 18:51:04 +02:00
Emil 0051af430b Put `application/json` as a higher preference than `application/octet-stream` 2022-01-30 18:50:14 +02:00
tersec 29e2169585
phase 0 & altair beacon chain and altair validator spec URL updates (#3339) 2022-01-29 13:53:31 +00:00
tersec 89ffa8a1a7
spec URL & copyright year update (#3338) 2022-01-29 01:05:39 +00:00
tersec 60bf5b8bf4
use v1.1.9 test vectors (#3337) 2022-01-28 22:47:48 +00:00
tersec 95fee10328
clean up hashed rollback proc declarations (#3333)
* clean up hashed rollback proc declarations

* use generic hashed rollback proc type
2022-01-28 14:24:37 +00:00
cheatfate 1287a20b13 Use HTTP status codes instead of status in body. 2022-01-28 15:36:27 +02:00
Jacek Sieka e264276b36
keep unviables in quarantine (#3331)
they remain unviable even after a reorg
2022-01-28 11:59:55 +01:00
Zahary Karadjov 49b7daa39d [ncli_db] bugfix: take into account finalization delay in reward calc post Altair
This fixes a problem affecting Prater's epoch 64444.
2022-01-28 12:03:23 +02:00
tersec dcb671617c
add/support TERMINAL_BLOCK_HASH_ACTIVATION_EPOCH (#3303) 2022-01-27 19:52:08 +00:00
Jacek Sieka 84b6ad871d harden status message handling
Additional sanity checking of the status message exchanged during a
fresh connection:

* check that head and finalized make sense, slot-wise
* verify that finalized root lies on the canonical chain, when possible
* re-check these things for every status message during sync
2022-01-27 18:46:47 +02:00
Eugene Kabanov aa27baacf5
Fix 408 Timeout error returned by REST server. (#3301)
* Disable REST server timeouts.
* Add options to CLI to tune REST server parameters.
2022-01-27 18:41:05 +02:00
tersec 7c51da037f
add block gossip validation condition (#3325) 2022-01-26 17:22:06 +00:00
tersec 2b4a960270
rename On{Merge,Bellatrix}BlockAdded and Rollback{Merge,Bellatrix}HashedProc (#3321) 2022-01-26 13:21:29 +01:00
Jacek Sieka f70aceef37
Harden handling of unviable forks (#3312)
* Harden handling of unviable forks

In our current handling of unviable forks, we allow peers to send us
blocks that come from a different fork - this is not necessarily an
error as it can happen naturally, but it does open up the client to a
case where the same unviable fork keeps getting requested - rather than
allowing this to happen, we'll now give these peers a small negative
score - if it keeps happening, we'll disconnect them.

* keep track of unviable forks in quarantine, to avoid filling it with
known junk
* collect peer scores in single module
* descore peers when they send unviable blocks during sync
* don't give score for duplicate blocks
* increase quarantine size to a level that allows finality to happen
under optimal conditions - this helps avoid downloading the same blocks
over and over in case of an unviable fork
* increase initial score for new peers to make room for one more failure
before disconnection
* log and score invalid/unviable blocks in requestmanager too
* avoid ChainDAG dependency in quarantine
* reject gossip blocks with unviable parent
* continue processing unviable sync blocks in order to build unviable
dag

* docs

* Update beacon_chain/consensus_object_pools/block_pools_types.nim

* add unviable queue test
2022-01-26 13:20:08 +01:00
tersec bd0a3a9b10
rearrange MEV code (#3319) 2022-01-25 19:43:28 +00:00
Emil efbd939108 Make `handleValidatorExitCommand` work with `REST API` 2022-01-25 14:00:29 +02:00
Jacek Sieka d076e1a11b
ncli_db: import states and blocks from era file (#3313) 2022-01-25 09:28:26 +01:00
tersec 00a347457a
dynamic sync committee subscriptions (#3308)
* dynamic sync committee subscriptions

* fast-path trivial case rather than rely on RNG with probability 1 outcome

Co-authored-by: zah <zahary@gmail.com>

* use func instead of template; avoid calling async function unnecessarily

* avoid unnecessary sync committee topic computation; use correct epoch lookahead; enforce exception/effect tracking

* don't over-optimistically update ENR syncnets; non-looping version of nearSyncCommitteePeriod

* allow separately setting --allow-all-{sub,att,sync}nets

* remove unnecessary async

Co-authored-by: zah <zahary@gmail.com>
2022-01-24 20:40:59 +00:00
tersec 062275461c
add flashbots (milestone 1) consensus beacon block types (#3314)
* add flashbots (milestone 1) consensus beacon block types

* remove MEV types from main bellatrix spec module
2022-01-24 20:15:22 +00:00
tersec 351c2fd48a
rename mergeData to bellatrixData and mergeFork to bellatrixFork (#3315) 2022-01-24 16:23:13 +00:00
Jacek Sieka 61342c2449
limit by-root requests to non-finalized blocks (#3293)
* limit by-root requests to non-finalized blocks

Presently, we keep a mapping from block root to `BlockRef` in memory -
this has simplified reasoning about the dag, but is not sustainable with
the chain growing.

We can distinguish between two cases where by-root access is useful:

* unfinalized blocks - this is where the beacon chain is operating
generally, by validating incoming data as interesting for future fork
choice decisions - bounded by the length of the unfinalized period
* finalized blocks - historical access in the REST API etc - no bounds,
really

In this PR, we limit the by-root block index to the first use case:
finalized chain data can more efficiently be addressed by slot number.

Future work includes:

* limiting the `BlockRef` horizon in general - each instance is 40
bytes+overhead which adds up - this needs further refactoring to deal
with the tail vs state problem
* persisting the finalized slot-to-hash index - this one also keeps
growing unbounded (albeit slowly)

Anyway, this PR easily shaves ~128mb of memory usage at the time of
writing.

* No longer honor `BeaconBlocksByRoot` requests outside of the
non-finalized period - previously, Nimbus would generously return any
block through this libp2p request - per the spec, finalized blocks
should be fetched via `BeaconBlocksByRange` instead.
* return `Opt[BlockRef]` instead of `nil` when blocks can't be found -
this becomes a lot more common now and thus deserves more attention
* `dag.blocks` -> `dag.forkBlocks` - this index only carries unfinalized
blocks from now - `finalizedBlocks` covers the other `BlockRef`
instances
* in backfill, verify that the last backfilled block leads back to
genesis, or panic
* add backfill timings to log
* fix missing check that `BlockRef` block can be fetched with
`getForkedBlock` reliably
* shortcut doppelganger check when feature is not enabled
* in REST/JSON-RPC, fetch blocks without involving `BlockRef`

* fix dag.blocks ref
2022-01-21 13:33:16 +02:00
tersec 1a37cae329
allow Eth1 monitor to run without genesis_deposit_contract_snapshot.ssz (#3279) 2022-01-21 12:59:09 +02:00
Eugene Kabanov 0ea6dfa517
Fix current slot value and finishing progress for backfilling. (#3304) 2022-01-21 10:35:54 +01:00
Dustin Brody 9699858422 rename MERGE_FORK_VERSION to BELLATRIX_FORK_VERSION 2022-01-20 19:33:05 +02:00
Mamy Ratsimbazafy 9e9ccf4a1f
Slashing prot interchange tests v5.2.1 (#3277)
* initial support for minification and new interchange tests. Removal of v1 and v1 migration.

* Synthetic attestations: SQLite3 requires one statement/query per prepared statement

* Fix DB import interrupted if no attestation was found

* Skip test relying on undocumented test behavior (https://github.com/eth-clients/slashing-protection-interchange-tests/pull/12#issuecomment-1011158701)

* Skip test relying on unclear minification behavior:
creating an invalid minified attestation with source > target or setting target = max(source, target)

* remove DB v1 and update submodule

* Apply suggestions from code review

Co-authored-by: Jacek Sieka <jacek@status.im>

Co-authored-by: Jacek Sieka <jacek@status.im>
2022-01-20 17:14:06 +01:00
Jacek Sieka 1df549143e
rest: fix GraffitiBytes serialization (#3299) 2022-01-20 13:31:55 +01:00
Jacek Sieka 570379d3d9
Backfiller (#3263)
Backfilling is the process of downloading historical blocks via P2P that
are required to fulfill `GetBlocksByRange` duties - this happens during
both trusted node and finalized checkpoint syncs.

In particular, backfilling happens after syncing to head, such that
attestation work can start as soon as possible.

* Fix SyncQueue initialization procedure.
Remove usage of `awaitne`.
Add cancellation support.
Remove unneeded `sleepAsync()` if peer's head is older than needed.
Add `direction` field to all logs.
Fix syncmanager wedge issue.
Add proper resource cleaning procedure on backward sync finish.

Co-authored-by: cheatfate <eugene.kabanov@status.im>
2022-01-20 08:25:45 +01:00
Jacek Sieka e4939538cd
fix non-totalized validator metric (#3297)
...when running on host with many validators, this explodes
2022-01-19 15:20:48 +01:00
tersec 2f635d3337
rename *_{MERGE => BELLATRIX} constant names (#3296) 2022-01-18 16:31:05 +00:00
tersec 9c0c9c98ce
complete switch to beacon_chain/specs/datatypes/bellatrix (#3295) 2022-01-18 13:36:52 +00:00
Zahary Karadjov 47f1f7ff1a More efficient reward data persistance; Address review comments
The new format is based on compressed CSV files in two channels:

* Detailed per-epoch data
* Aggregated "daily" summaries

The use of append-only CSV file speeds up significantly the epoch
processing speed during data generation. The use of compression
results in smaller storage requirements overall. The use of the
aggregated files has a very minor cost in both CPU and storage,
but leads to near interactive speed for report generation.

Other changes:

- Implemented support for graceful shut downs to avoid corrupting
  the saved files.

- Fixed a memory leak caused by lacking `StateCache` clean up on each
  iteration.

- Addressed review comments

- Moved the rewards and penalties calculation code in a separate module

Required invasive changes to existing modules:

- The `data` field of the `KeyedBlockRef` type is made public to be used
  by the validator rewards monitor's Chain DAG update procedure.

- The `getForkedBlock` procedure from the `blockchain_dag.nim` module
  is made public to be used by the validator rewards monitor's Chain DAG
  update procedure.
2022-01-18 01:56:56 +02:00
Zahary Karadjov 29aad0241b Precise per-component ETH-denominated rewards tracking
This is an alternative take on https://github.com/status-im/nimbus-eth2/pull/3107
that aims for more minimal interventions in the spec modules at the expense of
duplicating more of the spec logic in ncli_db.
2022-01-18 01:56:56 +02:00
Jacek Sieka 4e2d2ff7f4 rest: fix invalid type `RestSyncCommitteeSubscription`
Using the wrong type here causes requests to fail due to the overly
zealous parameter validation - the failure is harmless in the current
duty subscription model, but would have caused more serious failures
down the line.
2022-01-17 22:33:24 +02:00
Jacek Sieka 6bf3330d73
fix ugly delay logging 2022-01-17 20:12:36 +01:00
Jacek Sieka ff5b91cd58
Revert "Don't use GC memory for the initial beacon block summaries loading" (#3292)
This reverts commit 7e2fc2b726.
2022-01-17 12:07:49 +00:00
Jacek Sieka 836f6984bb
move `state_transition` to `Result` (#3284)
* better error messages in api
* avoid `BlockData` copies when replaying blocks
2022-01-17 12:19:58 +01:00
Jacek Sieka 68247f81b3
Trusted node sync (#3209)
* Trusted node sync

Trusted node sync, aka checkpoint sync, allows syncing tyhe chain from a
trusted node instead of relying on a full sync from genesis.

Features include:

* sync from any slot, including the latest finalized slot
* backfill blocks either from the REST api (default) or p2p (#3263)

Future improvements:

* top up blocks between head in database and some other node - this
makes for an efficient backup tool
* recreate historical state to enable historical queries

* fixes

* load genesis from network metadata
* check checkpoint block root against state
* fix invalid block root in rest json decoding
* odds and ends

* retry looking for epoch-boundary checkpoint blocks
2022-01-17 10:27:08 +01:00
Zahary Karadjov ebde027262 Re-enable the HTTP support in Eth1Monitor
This reverts commit 6fddff524c.
2022-01-16 18:26:21 +02:00
Zahary Karadjov 7e2fc2b726 Don't use GC memory for the initial beacon block summaries loading 2022-01-15 10:15:17 +02:00
Jacek Sieka 167068e739
valmon: fix `--validator-monitor-totals` feature (#3286)
Else log size explodes for machines with many validators
2022-01-14 15:57:46 +01:00
Zahary Karadjov bef13b6cce
Version 1.6.0 2022-01-14 13:52:06 +02:00
tersec d878948ed2
update sync committee gossip validation comments; spec URL updates (#3280) 2022-01-13 13:46:08 +00:00
Jacek Sieka d57c2dc4e5
use tail block as sync pivot (#3276)
When syncing, we show how much of the sync has completed - with
checkpoint sync, the syncing does not always go from slot 0 to head, but
rather can start in the middle.

To show a consistent `%` between restarts, we introduce the concept of a
pivot point, such that if I sync 10% of the chain, then restart the
client, it picks up at 10% (instead of counting from 0).

What it looks like:
```
INF ... sync="01d12h41m (15.96%) 13.5158slots/s (QDDQDDQQDP:339018)" ...
```
2022-01-13 10:37:53 +01:00
Jacek Sieka e9486f5e5b
state_sim: clean up attestation production (#3274)
* use same naming as everywhere
* avoid iterator bug that leads to state copy
2022-01-12 21:42:03 +01:00
tersec 14aab2c13f
update 10 modules from using merge to bellatrix (#3272) 2022-01-12 15:50:30 +01:00
Jacek Sieka 805e85e1ff
time: spring cleaning (#3262)
Time in the beacon chain is expressed relative to the genesis time -
this PR creates a `beacon_time` module that collects helpers and
utilities for dealing the time units - the new module does not deal with
actual wall time (that's remains in `beacon_clock`).

Collecting the time related stuff in one place makes it easier to find,
avoids some circular imports and allows more easily identifying the code
actually needs wall time to operate.

* move genesis-time-related functionality into `spec/beacon_time`
* avoid using `chronos.Duration` for time differences - it does not
support negative values (such as when something happens earlier than it
should)
* saturate conversions between `FAR_FUTURE_XXX`, so as to avoid
overflows
* fix delay reporting in validator client so it uses the expected
deadline of the slot, not "closest wall slot"
* simplify looping over the slots of an epoch
* `compute_start_slot_at_epoch` -> `start_slot`
* `compute_epoch_at_slot` -> `epoch`

A follow-up PR will (likely) introduce saturating arithmetic for the
time units - this is merely code moves, renames and fixing of small
bugs.
2022-01-11 11:01:54 +01:00
tersec ae61512ee9
rename upgrade_to_{merge,bellatrix}; detect unchanging spec YAMLs (#3265) 2022-01-10 09:39:43 +00:00
Jacek Sieka 20e700fae4
Harden CommitteeIndex, SubnetId, SyncSubcommitteeIndex (#3259)
* Harden CommitteeIndex, SubnetId, SyncSubcommitteeIndex

Harden the use of `CommitteeIndex` et al to prevent future issues by
using a distinct type, then validating before use in several cases -
datatypes in spec are kept simple though so that invalid data still can
be read.

* fix invalid epoch used in REST
`/eth/v1/beacon/states/{state_id}/committees` committee length (could
return invalid data)
* normalize some variable names
* normalize committee index loops
* fix `RestAttesterDuty` to use `uint64` for `validator_committee_index`
* validate `CommitteeIndex` on ingress in REST API
* update rest rules with stricter parsing
* better REST serializers
* save lots of memory by not using `zip` ...at least a few bytes!
2022-01-09 01:28:49 +02:00
Jacek Sieka 6f7e0e3393
REST cleanups (#3255)
* REST cleanups

* reject out-of-range committee requests
* print all hex values as lower-case
* allow requesting state information by head state root
* turn `DomainType` into array (follow spec)
* `uint_to_bytesXX` -> `uint_to_bytes` (follow spec)
* fix wrong dependent root in `/eth/v1/validator/duties/proposer/`
* update documentation - `--subscribe-all-subnets` is no longer needed
when using the REST interface with validator clients
* more fixes
* common helpers for dependent block
* remove test rules obsoleted by more strict epoch tests
* fix trailing commas

* Update docs/the_nimbus_book/src/rest-api.md
* Update docs/the_nimbus_book/src/rest-api.md

Co-authored-by: sacha <sacha@status.im>
2022-01-08 22:06:34 +02:00
Jacek Sieka 25bb927e62
better web3monitor error message (#3260)
```
WRN 2022-01-08 11:35:00.963+01:00 Eth1 chain monitoring failure,
restarting  topics="eth1" err="Failed to setup web3 connection: (111)
Connection refused"
```
2022-01-08 14:35:36 +01:00
tersec bac0eaa92e
update 10 modules from using merge to bellatrix (#3257) 2022-01-07 18:10:40 +01:00
Jacek Sieka ba99c8fe4f
update era file documentation / impl (#3226)
Overhaul of era files, including documentation and reference
implementations

* store blocks, then state, then slot indices for easy lookup at low
cost
* document era file rationale
* altair+ support in era writer
2022-01-07 11:13:19 +01:00
tersec 0fd8bf7b56
spec URL updates (#3254) 2022-01-06 18:35:38 +00:00
tersec 8242e57f41
initial migration from spec/datatypes/{merge => bellatrix} (#3249) 2022-01-06 12:25:35 +01:00
Emil 2a12d1c49f Support more content-types when specifying the Graffiti string 2022-01-06 10:56:59 +02:00
Jacek Sieka 0e2b4e39fa
REST JSON support improvements (#3232)
* support downloading blocks / states via JSON in addition to SSZ -
slow, but needed for infura support - SSZ is still used when server
supports it
* use common forked block/state reader in REST API
* fix stack overflows in REST JSON decoder
* fix invalid serialization of `justification_bits` in
`/eth/v1/debug/beacon/states` and `/eth/v2/debug/beacon/states`
* fix REST client to use `/eth/...` instead of `/api/eth/...`, update
"default" urls to expose REST api via `/eth` as well as this is what the
standard says - `/api` was added early on based on an example "base url"
in the spec that has been removed since
* expose Nimbus REST extensions via `/nimbus` in addition to
`/api/nimbus` to stay consistent with `/eth`
* fix invalid state root when reading states via REST
* fix recursive imports in `spec/ssz_codec`
* remove usages of `serialization.useCustomSerialization` - fickle
2022-01-06 08:38:40 +01:00
Jacek Sieka 0a4728a241
Handle access to historical data for which there is no state (#3217)
With checkpoint sync in particular, and state pruning in the future,
loading states or state-dependent data may fail. This PR adjusts the
code to allow this to be handled gracefully.

In particular, the new availability assumption is that states are always
available for the finalized checkpoint and newer, but may fail for
anything older.

The `tail` remains the point where state loading de-facto fails, meaning
that between the tail and the finalized checkpoint, we can still get
historical data (but code should be prepared to handle this as an
error).

However, to harden the code against long replays, several operations
which are assumed to work only with non-final data (such as gossip
verification and validator duties) now limit their search horizon to
post-finalized data.

* harden several state-dependent operations by logging an error instead
of introducing a panic when state loading fails
* `withState` -> `withUpdatedState` to differentiate from the other
`withState`
* `updateStateData` can now fail if no state is found in database - it
is also hardened against excessively long replays
* `getEpochRef` can now fail when replay fails
* reject blocks with invalid target root - they would be ignored
previously
* fix recursion bug in `isProposed`
2022-01-05 19:38:04 +01:00
zah fba1f08a5e
Implement #3129 (Optimized history traversals in the REST API) (#3219)
* Fix REST some rest call signatures and implement a simple API benchmark tool

* Implement #3129 (Optimized history traversals in the REST API)

Other notable changes:

The `updateStateData` procedure in the `blockchain_dag.nim` module is
optimized to not rewind down to the last snapshot state saved in the
database if the supplied input state can be used as a starting point
instead.

* Disallow await in withStateForBlockSlot
2022-01-05 15:49:10 +01:00
tersec 5878d34117
rename forkDigests.merge to forkDigests.bellatrix (#3245) 2022-01-05 14:24:15 +00:00
tersec 66c9b7fbce
shift block_sim fork epochs; allow VC to work with non-multiple-of-3 SECONDS_PER_SLOT (#3244) 2022-01-05 13:41:39 +00:00
tersec 7594fa660d
copyright year and spec URL updates (#3243) 2022-01-05 11:07:14 +00:00
Zahary Karadjov 54d0d588b1 Implementation of the Keymanager API (BETA)
https://github.com/ethereum/keymanager-APIs
2022-01-04 18:51:45 +02:00
Tanguy 511f2d24f0
Tune getLowSubnets (#3241)
* Tune getLowSubnets

* Also aim for dHigh peers in gossipsub

* Apply suggestions from code review

Co-authored-by: Jacek Sieka <jacek@status.im>

Co-authored-by: Jacek Sieka <jacek@status.im>
2022-01-04 14:37:04 +01:00
tersec b81c06edab
rename Beacon{Block,State}Fork.Merge to Bellatrix; update copyright years (#3240) 2022-01-04 09:45:38 +00:00
tersec d20387e910
update copyright years and spec URLs (#3239) 2022-01-04 06:08:19 +00:00
tersec da017d2ca5
update from phase0/altair v1.1.6 URLs to v1.1.8 spec URLs (#3238) 2022-01-04 03:57:15 +00:00
Jacek Sieka c4ce59e55b
Assorted logging improvements (#3237)
* log doppelganger detection when it activates and when it causes missed
duties
* less prominent eth1 sync progress
* log in-progress sync at notice only when actually missing duties
* better detail in replay log
* don't log finalization checkpoints - this is quite verbose when
syncing and already included in "Slot start"
2022-01-03 22:18:49 +01:00
tersec 3c63a78c01
use v1.1.8 test vectors (#3236) 2022-01-03 17:43:00 +00:00
Jacek Sieka 61b6fc1016
3x speedup in snappy compression (#3234)
* 3x speedup in snappy compression

oh, the wonders of `copyMem` in `endians2` - speeds up all kinds of
operations like database stores, sending gossip etc.

* endian usage fixes
2022-01-03 18:17:10 +01:00
tersec e78d12beb9
support GOSSIP_MAX_SIZE_MERGE blocks; prevent fork choice stutter via aggregate attestations (#3230)
* support GOSSIP_MAX_SIZE_MERGE-sized blocks; prevent fork choice clock stutter via aggregate attestations

* relay max gossip size to libp2p, use tight uncompressed bounds for fixed-size messages

* Update beacon_chain/networking/eth2_network.nim

Co-authored-by: Jacek Sieka <jacek@status.im>

* Update beacon_chain/networking/eth2_network.nim

Co-authored-by: Jacek Sieka <jacek@status.im>

Co-authored-by: Jacek Sieka <jacek@status.im>
2022-01-03 16:20:15 +00:00
tersec 8be1699014
use v1.1.7 test vectors (#3231)
* use v1.1.7 test vectors
2022-01-03 13:06:14 +00:00
tersec d4680df8d2
convert between engine and consensus ExecutionPayloads (#3228)
* convert between engine and consensus ExecutionPayloads
2022-01-03 13:22:56 +01:00