* Trusted node sync
Trusted node sync, aka checkpoint sync, allows syncing tyhe chain from a
trusted node instead of relying on a full sync from genesis.
Features include:
* sync from any slot, including the latest finalized slot
* backfill blocks either from the REST api (default) or p2p (#3263)
Future improvements:
* top up blocks between head in database and some other node - this
makes for an efficient backup tool
* recreate historical state to enable historical queries
* fixes
* load genesis from network metadata
* check checkpoint block root against state
* fix invalid block root in rest json decoding
* odds and ends
* retry looking for epoch-boundary checkpoint blocks
* min viable changes
* make clear keymanager api is not ready for mainnet
* Update docs/the_nimbus_book/src/keymanager-api.md
Co-authored-by: Jacek Sieka <jacek@status.im>
* Update docs/the_nimbus_book/src/keymanager-api.md
Co-authored-by: zah <zahary@gmail.com>
Co-authored-by: Jacek Sieka <jacek@status.im>
Co-authored-by: zah <zahary@gmail.com>
* REST cleanups
* reject out-of-range committee requests
* print all hex values as lower-case
* allow requesting state information by head state root
* turn `DomainType` into array (follow spec)
* `uint_to_bytesXX` -> `uint_to_bytes` (follow spec)
* fix wrong dependent root in `/eth/v1/validator/duties/proposer/`
* update documentation - `--subscribe-all-subnets` is no longer needed
when using the REST interface with validator clients
* more fixes
* common helpers for dependent block
* remove test rules obsoleted by more strict epoch tests
* fix trailing commas
* Update docs/the_nimbus_book/src/rest-api.md
* Update docs/the_nimbus_book/src/rest-api.md
Co-authored-by: sacha <sacha@status.im>
* Fix REST some rest call signatures and implement a simple API benchmark tool
* Implement #3129 (Optimized history traversals in the REST API)
Other notable changes:
The `updateStateData` procedure in the `blockchain_dag.nim` module is
optimized to not rewind down to the last snapshot state saved in the
database if the supplied input state can be used as a starting point
instead.
* Disallow await in withStateForBlockSlot
Validator monitoring based on and mostly compatible with the
implementation in Lighthouse - tracks additional logs and metrics for
specified validators so as to stay on top on performance.
The implementation works more or less the following way:
* Validator pubkeys are singled out for monitoring - these can be
running on the node or not
* For every action that the validator takes, we record steps in the
process such as messages being seen on the network or published in the
API
* When the dust settles at the end of an epoch, we report the
information from one epoch before that, which coincides with the
balances being updated - this is a tradeoff between being correct
(waiting for finalization) and providing relevant information in a
timely manner)
* Support starting from altair
* hide `finalized-checkpoint-` - they are incomplete and usage may cause
crashes
* remove genesis detection code (broken, obsolete)
* enable starting ChainDAG from altair checkpoints - this is a
prerequisite for checkpoint sync (TODO: backfill)
* tighten checkpoint state conditions
* show error when starting from checkpoint with existing database (not
supported)
* print rest-compatible JSON in ncli/state_sim
* altair/merge support in ncli
* more altair/merge support in ncli_db
* pre-load header to speed up loading
* fix forked block decoding
* Reduce the logging-related breaking changes
* Don't disable the stdout log when the `--log-file` option is used
* Rename `--log-stdout` to `--log-format` and hide it
* Deprecate the `--log-file` option
* Logging and startup improvements
Color support for released binaries!
* startup scripts no longer log to file by default - this only affects
source builds - released binaries don't support file logging
* add --log-stdout option to control logging to stdout (colors, json)
* detect tty:s vs redirected logs and log accordingly
* add option to disable log colors at runtime
* simplify several "common" logs, showing the most important information
earlier and more clearly
* remove line numbers / file information / tid - these take up space and
are of little use to end users
* still enabled in debug builds and tools
* remove `testnet_servers_image` compile-time option
* server images, released binaries and compile-from-source now offer
the same behaviour and features
* fixes https://github.com/status-im/nimbus-eth2/issues/2326
* fixes https://github.com/status-im/nimbus-eth2/issues/1794
* remove instanteneous block speed from sync message, keeping only
average
before:
```
INF 2021-10-28 16:45:59.000+02:00 Slot start topics="beacnde" tid=386429 file=nimbus_beacon_node.nim:884 lastSlot=2384027 wallSlot=2384028 delay=461us84ns peers=0 head=75a10ee5:3348 headEpoch=104 finalized=cd6804ba:3264 finalizedEpoch=102 sync="wwwwwwwwww:0:0.0000:0.0000:00h00m (3348)"
INF 2021-10-28 16:45:59.046+02:00 Slot end topics="beacnde" tid=386429 file=nimbus_beacon_node.nim:821 slot=2384028 nextSlot=2384029 head=75a10ee5:3348 headEpoch=104 finalizedHead=cd6804ba:3264 finalizedEpoch=102 nextAttestationSlot=-1 nextProposalSlot=-1 nextActionWait=n/a
```
after:
```
INF 2021-10-28 22:43:23.033+02:00 Slot start topics="beacnde" slot=2385815 epoch=74556 sync="DDPDDPUDDD:10:5.2258:01h19m (2361088)" peers=37 head=eacd2dae:2361096 finalized=73782:a4751487 delay=33ms687us715ns
INF 2021-10-28 22:43:23.291+02:00 Slot end topics="beacnde" slot=2385815 nextActionWait=n/a nextAttestationSlot=-1 nextProposalSlot=-1 head=eacd2dae:2361096
```
* fix comment
* documentation updates
* mention `--log-file` may be deprecated in the future
* update various docs
The developer documentation contains a comment mentioning the version of
Nim that the project is tested against. This comment was not updated for
a while (last Nim update in #2508). Bumping now to be in sync once more.
This refactoring puts the JSON-RPC and REST APIs on more equal footing
by renaming and moving things around, creating a separation between
client and server, and documenting what they are - the aim is to have a
simple-to-use base to start from when developing API clients, as well as
make it easier to navigate the code when looking for the legacy JSON-RPC
interface vs the new REST API.
* move REST client, serialization and supporting types to spec/eth2_apis
* REST stuff now starts with `rest_`, JSON-RPC stuff starts with `rpc_`,
more or less
* simplify imports such that there's a simple module to import for both
server and client
* map REST type and proc names to yaml spec more closely - in
particular, reuse operation and type names in `rest_types` to make
comparisons against spec more easy
* cleaner separation between client and server modules - modules common
between server and client such as `rest_types` and serialization move to
the spec folder - this allows the client to be built with less knowledge
about server internals
* cp
* rename API to JSON-RPC API
* rename api to json-rpc api, add rest api page
* remove extra emacs generated file
* remove extra emacs generated file
* add doppelganger detection explanation to FAQ and link to Joe's guide from our Pi guide
* Edit CLI options page
* cp
* merge with ith unstable
* prysm migration guide: add steps until import slashing db
* update systemd
* update troubleshooting
* update pi guide
* metrics: replace winners with community
* update migration guide
* cp
* progress with guides, various edits
* update nimbus book readme
* Revert "update nimbus book readme"
This reverts commit d568f0f7a3.
* update binaries page
* rm extra emacs generated files
* remove extra emacs generated file
* fix windows typ
* fix eth1 title
* page on adding additional validator
* Add default port info
* Add default port info
* Edit CLI and docs to reflect 9000 port
* Apply suggestions from code review
Co-authored-by: Jacek Sieka <arnetheduck@gmail.com>
* fix systemd guide typo
* Update update flow
* update Pi guide: sync speed estimates
* FAQ: wait two epochs to prevent slashing
* Add to syncing logs
* Pi guide: add link to USB3.0 issue and link to recommended workaround
* Pi guide edits
* Calculate how much time syncing takes, and show it at statusbar.
* Pi edits
* add 64-bit check to pi guide
* minor edit
Co-authored-by: cheatfate <eugene.kabanov@status.im>
* update ve1.0.0-rc.0 preset spec references
* remove runtime preset ETH1_FOLLOW_DISTANCE from preset files; remove two CI build items to try to keep Travis from timing out
add section addressing `address already in use error`. we already sort of cover this in the guide, but i think it makes sense to have it explicitly mentioned here.
* book updates
* save progress
* save progress
* work on deposit data section + edits
* title edit
* minor edits
* edits and clarifications
* add images, launchpad steps, and a note on expected wait time
* add images + edits
* restructure, simplify, edits
* faq corrections
* rework intro: cp
* update intro and faq
* add note on requestiong goerli eth
* typo fix
* add numbers to steps in walletstore guide"
* chapter 8 title: replace Nimbus with NBC
* Update docs/the_nimbus_book/src/medalla.md
Co-authored-by: Jacek Sieka <jacek@status.im>
Co-authored-by: Jacek Sieka <jacek@status.im>
* book updates
* save progress
* save progress
* work on deposit data section + edits
* title edit
* minor edits
* edits and clarifications
* add images, launchpad steps, and a note on expected wait time
* add images + edits
* restructure, simplify, edits
* faq corrections
* rework intro: cp
* update intro and faq
* add note on requestiong goerli eth
* typo fix
* add numbers to steps in walletstore guide"
* chapter 8 title: replace Nimbus with NBC
* Update docs/the_nimbus_book/src/medalla.md
Co-authored-by: Jacek Sieka <jacek@status.im>
Co-authored-by: Jacek Sieka <jacek@status.im>
- testnets can now be launched with a separate validator client - make altona SCRIPT_PARAMS="--separateVC"
- reverted the ctrl+C signal handler code reuse - not necessary for the VC anyway (default is good enough)
- added a bit more logging in the VC
- removed unnecessary code in the VC - connect() just parses the address & port...
- fixed a couple more VC issues - when fetching the duties for an epoch fails on the BN side ==> the VC shouldn't be left in a broken state
- documented the currently supported json-rpc endpoints
- added more checks on the BN side for the API - bounds-checking the requests & also checking if the BN itself is synced
- other cleanup
currently a local sim doesn't finalize, but participation in the altona network with a separate VC is painless and works just as well as with in-process validators in a BN
Jacek Sieka
sacha
Zahary Karadjov
rasberrylasagna
tersec
Ștefan Talpalaru
Etan Kissling
Zachinquarantine
Johann Bauer
yslcrypto
TennisBowling
Mentor Palokaj
Afr Schoe
Jakub Sokołowski
Nikola Ristić
Eugene Kabanov
Viktor Kirilov
Kim De Mey
Mamy André-Ratsimbazafy
Michael Bradley