Commit Graph

498 Commits

Author SHA1 Message Date
tersec 76b3db3d7e
add prater.beaconcha.in external block builder overview link (#4211)
* add prater.beaconcha.in external block builder overview link

* Update docs/the_nimbus_book/src/external-block-builder.md

Co-authored-by: Etan Kissling <etan@status.im>

Co-authored-by: Etan Kissling <etan@status.im>
2022-10-03 16:56:49 +00:00
Julian Ste 86800066e4
Typo and updated specific version (#4203)
* Typo and updated specific version

* Update keep-updated.md

* Update keep-updated.md
2022-10-01 20:20:11 +00:00
Julian Ste bee50d7578
Updated nomenclature for eth2 naming (#4202) 2022-10-01 20:01:41 +00:00
tersec 6c18e82506
link to document of available external builder relays (#4157) 2022-09-26 19:11:34 +00:00
tersec 3c03ba86c1
update consensus spec ref URLs to v1.2.0 (#4164) 2022-09-23 07:56:06 +00:00
Etan Kissling 0708fcd7cf
rm require engine API check (#4144)
The `eth1_monitor` check to require engine API from bellatrix onward
has issues in setups where the EL and CL are started simultaneously
because the EL may not be ready to answer requests by the time that the
check is performed. This can be observed, e.g., on Raspberry Pi 4 when
using Besu as the EL client. Now that the merge transition happened, the
check is also not that useful anymore, as users have other ways to know
that their setup is not working correctly (e.g., repeated exchange logs)
2022-09-19 23:47:46 +02:00
Kim De Mey c607d7da0b
Trusted node sync clarifications in nimbus book (#4146)
- Make it clear that it is a separate command from actually
running the beacon node
- Fix and extend the note on starting from a clean folder
2022-09-19 13:32:28 +03:00
Shiti Saxena dc21897e48
docs: add note about data-dir (#4124) 2022-09-17 08:03:38 +03:00
tersec 5677816df3
remove obsolete merge test vector test bash script (#4135)
* remove obsolete merge test vector test bash script

* remove reference to removed merge test vector script
2022-09-16 22:49:39 +02:00
Jacek Sieka 23fb596779
add link to eth-clients checkpoint sync sources 2022-09-15 13:02:05 +02:00
Miguel Tenorio f935f3527b
Fix typo in docs/docker.md (#4110) 2022-09-10 18:50:57 +02:00
Jacek Sieka c74b6040eb
docs: fix date 2022-09-06 08:25:34 +02:00
Jacek Sieka dd0a14aa2e
merge docs update
Note the date and time
2022-09-06 07:50:56 +02:00
omahs 2ad5239fae
Fix: typos (#4064)
* Fix: typos

Fix: typos

* Fix: typo

Fix: typo

* Fix: typos

Fix: typos

* Fix: typo

Fix: typo

* Fix: typos

Fix: typos

* Fix: typo

Fix: typo
2022-09-05 14:25:02 +00:00
Etan Kissling 613f4a9a50
accelerate EL sync with LC with `--sync-light-client` (#4041)
When the BN-embedded LC makes sync progress, pass the corresponding
execution block hash to the EL via `engine_forkchoiceUpdatedV1`.
This allows the EL to sync to wall slot while the chain DAG is behind.
Renamed `--light-client` to `--sync-light-client` for clarity, and
`--light-client-trusted-block-root` to `--trusted-block-root` for
consistency with `nimbus_light_client`.

Note that this does not work well in practice at this time:
- Geth sticks to the optimistic sync:
  "Ignoring payload while snap syncing" (when passing the LC head)
  "Forkchoice requested unknown head" (when updating to LC head)
- Nethermind syncs to LC head but does not report ancestors as VALID,
  so the main forward sync is still stuck in optimistic mode:
  "Pre-pivot block, ignored and returned Syncing"

To aid EL client teams in fixing those issues, having this available
as a hidden option is still useful.
2022-08-29 12:16:35 +00:00
tersec d7e9c334ac
document external block builder configuration (#4032)
* document external block builder configuration

* Update docs/the_nimbus_book/src/external-block-builder.md

Co-authored-by: Jacek Sieka <jacek@status.im>

* unhide external payload builder options

* clarify builder API incentive misalignment

Co-authored-by: Jacek Sieka <jacek@status.im>
2022-08-29 12:59:12 +03:00
Jacek Sieka addf55e075
update service file link (fixes #4035) 2022-08-26 14:16:38 +02:00
Etan Kissling 9180f09641
reduce LC optsync latency (#4002)
The optimistic sync spec was updated since the LC based optsync module
was introduced. It is no longer necessary to wait for the justified
checkpoint to have execution enabled; instead, any block is okay to be
optimistically imported to the EL client, as long as its parent block
has execution enabled. Complex syncing logic has been removed, and the
LC optsync module will now follow gossip directly, reducing the latency
when using this module. Note that because this is now based on gossip
instead of using sync manager / request manager, that individual blocks
may be missed. However, EL clients should recover from this by fetching
missing blocks themselves.
2022-08-25 03:53:59 +00:00
zah 4e41ed1d5a
Require properly configured Engine API connection after the merge (#4006) 2022-08-22 22:44:40 +03:00
Etan Kissling 74dc388ad9
do not prune LC data by default (#4008)
Aligns the default retention policy for LC data with the one for blocks.
Minimum spec requirement for both blocks and LC data is ~5 months.
Additional use cases are better supported by retaining data for longer.
2022-08-21 11:24:59 +02:00
tersec c65eaca1bf
update spec ref URLs (#4005) 2022-08-20 16:03:32 +00:00
tersec 8274d5373b
update spec ref URLs (#3979) 2022-08-17 11:33:19 +00:00
Jacek Sieka 44ea08240c
web3 doc updates (#3971) 2022-08-17 13:37:25 +03:00
zah ca3245c4f0
Doppelganger exit code changed from 1031 to 129 (addresses #3973) (#3977) 2022-08-17 08:13:55 +02:00
Jacek Sieka 4adf849d09
book edits (#3955)
* reorganise navigation menus
* update light client guide with comparison table
* add suggested fee recipient and JWT secrets to the merge guide
* add some background info to book readme
* add JWT docs

also limit toc to make it displayable with substeps.
2022-08-13 20:17:20 +02:00
Etan Kissling 9c6a4316aa
document LC data serving options (#3922)
Adds a documentation page for configuring LC data serving.
2022-08-02 12:23:03 +00:00
tersec 73900172cb
update Goerli/Prater documentation and remove Kiln documentation (#3925) 2022-08-01 09:52:36 +03:00
tersec a8cefee4c5
remove obsolete merge documentation (#3923) 2022-07-29 18:02:09 +00:00
Etan Kissling 3ec7982293
update light client protocol version (#3550)
* Use final `v1` version for light client protocols
* Unhide LC data collection options
* Default enable LC data serving
* rm unneeded import
* Connect to EL on startup
* Add docs for LC based EL sync
2022-07-29 11:45:39 +03:00
Jacek Sieka dcea09ea5c
book edits (#3903)
* Reorganise menu, putting tabs on top
* Use tabs for specific content
* cleanups and consistency edits
2022-07-24 22:13:47 +02:00
Jacek Sieka dbd3d02e63
Migrate docs to mkdocs (#3900)
`mkdocs` works with markdown similar to `mdbook` but is generally more
pleasing to the eye and has several nice UX features.

This PR does the bulk of the transition - likely, a followup would be
needed to fully make use of the extra features and navigation.

Book pages have been kept url-compatible, meaning that for the most
part, old links should continue to work!

Co-authored-by: Etan Kissling <etan@status.im>
2022-07-22 21:47:24 +02:00
Jacek Sieka ddb7a3ecf0
polish intro 2022-07-22 18:18:31 +02:00
Jacek Sieka f98e9ec8bc
update docs (#3890)
* update docs

* introduce mdbook-admonish for nice looking callouts
* new section on data directory
* recommend source build for advanced users and direct the rest to
binaries
* more strongly highlight that execution client is needed
* write an actual deposit guide
* remove cruft / fix links / etc
2022-07-21 21:19:47 +03:00
Miran 76fa4772fb
fix a typo in a link (#3897)
Before the fix, the link would take you to the non-existing
https://nimbus.guide/prater.beaconcha.in
2022-07-21 21:08:37 +03:00
tersec e0e7af7aff
MEV boost script for ropsten and sepolia builder network (#3851)
* MEV boost script for ropsten and sepolia builder network

* networkInfo not a global variable
2022-07-12 20:50:12 +03:00
tersec ee27274078
update Kiln book chapter (#3853)
* update Kiln book chapter

* remove reference to kiln branch of nethermind
2022-07-12 07:51:40 +00:00
tersec 35712e0dd0
suggested fee recipient documentation (#3729)
* tentative suggested fee recipient description

* numbered priority list

* add chapter to book
2022-07-07 09:25:51 +00:00
Jacek Sieka ff12c7f9ce
remove web3 url prompt in launcher script (#3810)
* remove web3 url prompt in launcher script

The interactive prompt for web3 has outlived its utility as we now load
url:s from command line params and config files, preventing the prompt
from correctly detecting when it's needed.

Also, after the merge, a JWT secret will (likely) be needed.

* log notice when web3 url is missing
* fix docs to not mention default that doesn't exist
* fix scripts to properly quote arguments
2022-06-28 23:49:52 +03:00
Jacek Sieka 8b6f05a146
vc: initial documentation (#3798)
* vc: initial documentation

bare-bones, but enough to get started

* fixup

* Apply suggestions from code review

* remove prater, link upgrade

* reword
2022-06-27 17:24:55 +03:00
Jacek Sieka b3caacd59d
Doc updates (#3771)
* Doc updates

* initial merge guide
* mark JSON-RPC as removed
* expand on binaries
* recommend `-j4` when building
* cleanups

* style fixes
2022-06-19 10:24:01 +03:00
Jacek Sieka c48ea3cdc8
book updates (#3763)
* systemd: use same service file as package
* systemd: recommend setting up a user
* clean up obsolete / ephemeral notes
* eth1 -> execution, in many places
* clean up docker instructions to match the experience of running the
runner script
* add basic instructions / links for execution clients other than geth
* clear a few more references to JSON-RPC, update to REST
* update hardware requirements with execution client extras
* start talking about merge requirements (this needs a dedicated page)
* note that binaries do exist, where relevant
* list gitcoin under donations
* update syncing docs
* `NOT` is for `NOTICE` in logs
* update rpi hardware recommendations
* prevent automatic doppelganger restarts in systemd service file
2022-06-17 18:28:13 +03:00
zah e8efc0f184
Add support for the Sepolia network (#3762) 2022-06-16 17:11:26 +03:00
tersec cc5f95dbbb
separate non-zero exit code for doppelganger detection (#3728) 2022-06-10 14:53:19 +03:00
tersec 1262e50fa5
document MEV/mergemock test script (#3694) 2022-06-06 16:55:02 +03:00
Jacek Sieka 6a0433cb08
REST parameter defaults (#3689)
* add default BN for VC
* unify "default" beacon node across commands
* clean up validator exit docs
2022-06-01 10:47:52 +00:00
tersec a3413963a1
update (or for one, remove) 15 CL spec ref URLs (#3671) 2022-05-30 12:24:43 +00:00
tersec 92c1e75915
stop using --rpc in run-beacon-node and simulations (#3662) 2022-05-24 16:21:55 +00:00
tersec b3d603f364
more CL spec URL updates to v1.2.0-rc.1 (#3657) 2022-05-24 08:26:35 +00:00
0kok0 bb7662df3a
Security note (#3646) 2022-05-20 18:28:00 +03:00
zah e7ce3cacd0
Add support for the Ropsten beacon chain (#3648) 2022-05-20 18:26:07 +03:00
Nikolay Mitev b3dc9e2c08
Update merge test vector script and documentation (#3638) 2022-05-16 16:12:32 +03:00
Jacek Sieka 011e0ca02f
era file verification (#3605)
* era file verification

Implement and document era file verification

* era file states now come with block applied for easier verification
* clarify conflicting version handling
* document verification requirements
* remove count from name, use start-era, end-root to discover range

* remove obsolete todo

* abstract out block root loading
2022-05-10 03:28:46 +03:00
sacha c8260a4f15
add kiln instructions (do not merge WIP) (#3590)
* add kiln instructions

* update toc

* incorporate dustin's feedback modulo geth
2022-04-18 12:23:26 +02:00
Etan Kissling 745bf474a3
fix typo in e2store docs (#3602)
`BeackBlock` -> `BeaconBlock` typo.
2022-04-16 15:35:30 +00:00
Jacek Sieka f70ff38b53
enable `styleCheck:usages` (#3573)
Some upstream repos still need fixes, but this gets us close enough that
style hints can be enabled by default.

In general, "canonical" spellings are preferred even if they violate
nep-1 - this applies in particular to spec-related stuff like
`genesis_validators_root` which appears throughout the codebase.
2022-04-08 16:22:49 +00:00
Jacek Sieka f20c53be82
Revert "Polish, remove beta warning for validator monitor (#3531)" (#3578)
This reverts commit 540fc5a1af.
2022-04-08 15:05:17 +00:00
Jacek Sieka 540fc5a1af
Polish, remove beta warning for validator monitor (#3531)
Validator monitoring improves logging by giving more specific monitoring
information, and can now be seen as complete.

Previously, logging has focused on "Attestation sent" messages which
carry little informational value when things go wrong, and are overly
aggressive when everything works as expected (sending attestations is
the norm).

* lower "Attestation sent" log to `INFO`
* mark 1.7.0 as the start of the validator monitor feature - previous
versions had significant bugs in totals mode
2022-04-06 19:44:03 +03:00
Jacek Sieka 30eef0a369
Validator monitor polish (#3569)
* lower "Previous epoch attestation missing" to `NOTICE` for easier
filtering
* add delay logging to validator monitor logs
* simplify delay logging code post-`BeaconTime`
2022-04-06 09:23:01 +00:00
Jacek Sieka 6f4fa32c1d
remove bundled pyrmont support (#3568)
The `pyrmont` testnet has been discontinued.

For experiments, it's still possible to run pyrmont nodes by passing a
genesis/config, but this PR removes the bundled `--network:pyrmont`
option.

* update docs
* remove empty docs
* remove obsolete `eth2-stats` page
2022-04-05 12:59:25 +00:00
Jacek Sieka 4207b127f9
era: load blocks and states (#3394)
* era: load blocks and states

Era files contain finalized history and can be thought of as an
alternative source for block and state data that allows clients to avoid
syncing this information from the P2P network - the P2P network is then
used to "top up" the client with the most recent data. They can be
freely shared in the community via whatever means (http, torrent, etc)
and serve as a permanent cold store of consensus data (and, after the
merge, execution data) for history buffs and bean counters alike.

This PR gently introduces support for loading blocks and states in two
cases: block requests from rest/p2p and frontfilling when doing
checkpoint sync.

The era files are used as a secondary source if the information is not
found in the database - compared to the database, there are a few key
differences:

* the database stores the block indexed by block root while the era file
indexes by slot - the former is used only in rest, while the latter is
used both by p2p and rest.
* when loading blocks from era files, the root is no longer trivially
available - if it is needed, it must either be computed (slow) or cached
(messy) - the good news is that for p2p requests, it is not needed
* in era files, "framed" snappy encoding is used while in the database
we store unframed snappy - for p2p2 requests, the latter requires
recompression while the former could avoid it
* front-filling is the process of using era files to replace backfilling
- in theory this front-filling could happen from any block and
front-fills with gaps could also be entertained, but our backfilling
algorithm cannot take advantage of this because there's no (simple) way
to tell it to "skip" a range.
* front-filling, as implemented, is a bit slow (10s to load mainnet): we
load the full BeaconState for every era to grab the roots of the blocks
- it would be better to partially load the state - as such, it would
also be good to be able to partially decompress snappy blobs
* lookups from REST via root are served by first looking up a block
summary in the database, then using the slot to load the block data from
the era file - however, there needs to be an option to create the
summary table from era files to fully support historical queries

To test this, `ncli_db` has an era file exporter: the files it creates
should be placed in an `era` folder next to `db` in the data directory.
What's interesting in particular about this setup is that `db` remains
as the source of truth for security purposes - it stores the latest
synced head root which in turn determines where a node "starts" its
consensus participation - the era directory however can be freely shared
between nodes / people without any (significant) security implications,
assuming the era files are consistent / not broken.

There's lots of future improvements to be had:

* we can drop the in-memory `BlockRef` index almost entirely - at this
point, resident memory usage of Nimbus should drop to a cool 500-600 mb
* we could serve era files via REST trivially: this would drop backfill
times to whatever time it takes to download the files - unlike the
current implementation that downloads block by block, downloading an era
at a time almost entirely cuts out request overhead
* we can "reasonably" recreate detailed state history from almost any
point in time, turning an O(slot) process into O(1) effectively - we'll
still need caches and indices to do this with sufficient efficiency for
the rest api, but at least it cuts the whole process down to minutes
instead of hours, for arbitrary points in time

* CI: ignore failures with Nim-1.6 (temporary)

* test fixes

Co-authored-by: Ștefan Talpalaru <stefantalpalaru@yahoo.com>
2022-03-23 09:58:17 +01:00
Ștefan Talpalaru ea5c052016
enable multithreading by default (10-20% faster sync) (#3493) 2022-03-19 08:59:10 +01:00
tersec 8fbcf29775
update unchanged specs/phase0/p2p-interface.md URL references from v1.1.9 to v1.1.10 (#3510) 2022-03-16 10:40:35 +00:00
Dustin Brody 346407ef1c
running Nimbus on Kiln 2022-03-13 19:39:11 +00:00
Jacek Sieka d0183ccd77
Historical state reindex for trusted node sync (#3452)
When performing trusted node sync, historical access is limited to
states after the checkpoint.

Reindexing restores full historical access by replaying historical
blocks against the state and storing snapshots in the database.

The process can be initiated or resumed at any point in time.
2022-03-11 12:49:47 +00:00
zah 2dd02e54bc
Update the options page in the guide with explanation regarding the format of the config file (#3475) 2022-03-10 17:02:46 +02:00
Zahary Karadjov 13b264d509
Final v22.3.0: Add examples for migrating from JSON-RCP to REST 2022-03-10 16:46:11 +02:00
Zahary Karadjov 7340e7cab9
v22.3.0
* Deprecates the JSON-RPC API
2022-03-07 21:49:12 +02:00
TennisBowling 187eabeeb9
Docs fix (#3397)
* geth is bigger

* Update preparation.md

* Update install.md

* Update start-syncing.md

* Update docs/the_nimbus_book/src/start-syncing.md

Co-authored-by: Jacek Sieka <arnetheduck@gmail.com>

* Update install.md

* Update docs/the_nimbus_book/src/hardware.md

Co-authored-by: sacha <sacha.saint-leger@ethereum.org>

* Update docs/the_nimbus_book/src/install.md

Co-authored-by: sacha <sacha.saint-leger@ethereum.org>

* Update docs/the_nimbus_book/src/install.md

Co-authored-by: sacha <sacha.saint-leger@ethereum.org>

* Update docs/the_nimbus_book/src/install.md

Co-authored-by: sacha <sacha.saint-leger@ethereum.org>

* Update docs/the_nimbus_book/src/start-syncing.md

Co-authored-by: sacha <sacha.saint-leger@ethereum.org>

* Update docs/the_nimbus_book/src/preparation.md

Co-authored-by: sacha <sacha.saint-leger@ethereum.org>

* Update docs/the_nimbus_book/src/preparation.md

Co-authored-by: sacha <sacha.saint-leger@ethereum.org>

Co-authored-by: Jacek Sieka <arnetheduck@gmail.com>
Co-authored-by: sacha <sacha.saint-leger@ethereum.org>
2022-03-06 08:40:17 +01:00
tersec 35f712e378
kiln v2 documentation and testing (#3446) 2022-03-05 16:40:58 +02:00
tersec f0ada15dac
automated CL spec ref URL updates from v1.1.9 to v1.1.10 (#3455) 2022-03-02 10:00:21 +00:00
tersec 05bc61b712
add mev-boost RPC test, with docs (#3430)
* bump nim-web3 and add mev-boost RPC test, with docs

* remove trailing space

* use specific commithash
2022-02-24 14:38:31 +01:00
Paweł Bylica af3c65a689
Fix typo "step 3" -> "step 2" in migration.md (#3404) 2022-02-17 14:29:12 +00:00
sacha e81e67fe65
book updates (#3388) 2022-02-14 12:39:10 +01:00
Dustin Brody 3daa52ab87
update docs for geth/kiln 2022-02-11 20:06:06 +00:00
Jeremy Schlatter 47b1870100
update prysm export docs (#3365)
This command was [recently renamed](https://github.com/prysmaticlabs/prysm/pull/9873).
2022-02-09 11:25:12 +01:00
yslcrypto 71d98a03b3 add warning to REST API page 2022-02-08 18:30:59 +01:00
tersec de0d473ea1
docs: don't rely on ncli_db pruneDatabase for reducing storage usage (#3355)
* don't rely on ncli_db pruneDatabase for reducing storage usage

* remove "Running out of storage" section altogether

* Update docs/the_nimbus_book/src/troubleshooting.md

Co-authored-by: sacha <sacha@status.im>

Co-authored-by: sacha <sacha@status.im>
2022-02-03 12:35:34 +00:00
tersec 8e6a920bf4
rename MERGE_FORK_EPOCH to BELLATRIX_FORK_EPOCH (#3350)
* rename MERGE_FORK_EPOCH to BELLATRIX_FORK_EPOCH

* fix REST test rules
2022-02-02 14:06:55 +01:00
Zachinquarantine f5de887df7
Delete Pyrmont docs (#3340)
* Delete pyrmont.md

* Update log-rotate.md

* Update pi-guide.md
2022-02-01 12:05:20 +01:00
sacha 7d731322b2
Book: trusted sync (edits and clarifications) (#3329)
* edits

* more edits and clarifications

* edit

* add clarification on --trusted-node-url

* address feedback

* remove repetition
2022-02-01 12:02:04 +01:00
Jacek Sieka 3df9ffca9f val-mon: remove redundant `_total` suffix from counters
It turns out nim-metrics adds this suffix on its own - it also turns out
some of the names are non-conventional and need follow-up.
2022-01-31 18:51:24 +02:00
tersec 29e2169585
phase 0 & altair beacon chain and altair validator spec URL updates (#3339) 2022-01-29 13:53:31 +00:00
tersec 89ffa8a1a7
spec URL & copyright year update (#3338) 2022-01-29 01:05:39 +00:00
Jacek Sieka 5dd362fc6e
book: add trusted node sync to index (#3326)
* book: add trusted node sync to index

...and some doc updates

* Update docs/the_nimbus_book/src/start-syncing.md

Co-authored-by: Ștefan Talpalaru <stefantalpalaru@yahoo.com>

Co-authored-by: Ștefan Talpalaru <stefantalpalaru@yahoo.com>
2022-01-27 10:19:13 +01:00
sacha ed38b187cc
book: migration guide update (#3316)
* rest api: first draft

* minor edit

* add lighthouse export

* add teku

* cp

* cp

* fix merge conflict"

* fix typo

* incorporate feedback

* correct path mistake
2022-01-25 15:16:50 +01:00
sacha e07044754a
Update rest-api.md 2022-01-24 15:06:35 +01:00
sacha b7b517d97e
improve rest api documentation: first draft (#3306)
* rest api: first draft

* minor edit

* Update docs/the_nimbus_book/src/rest-api.md

Co-authored-by: Jacek Sieka <jacek@status.im>

* Update docs/the_nimbus_book/src/rest-api.md

Co-authored-by: Jacek Sieka <jacek@status.im>

* Update docs/the_nimbus_book/src/rest-api.md

Co-authored-by: Jacek Sieka <jacek@status.im>

* incorporate feedback

Co-authored-by: Jacek Sieka <jacek@status.im>
2022-01-24 14:46:40 +01:00
lbly 57abb07e4e
Update start-syncing.md (#3300)
I found this documentation a little confusing because it says "you must" leading me to think there may be some kind of dire consequences for NOT having it synced before starting the validator (in the context of importing existing keys from another setup, not starting a new validator with a new deposit).  Modifying the language to make the consequences of not having the syncing done clear.
2022-01-20 17:15:09 +01:00
Jacek Sieka 68247f81b3
Trusted node sync (#3209)
* Trusted node sync

Trusted node sync, aka checkpoint sync, allows syncing tyhe chain from a
trusted node instead of relying on a full sync from genesis.

Features include:

* sync from any slot, including the latest finalized slot
* backfill blocks either from the REST api (default) or p2p (#3263)

Future improvements:

* top up blocks between head in database and some other node - this
makes for an efficient backup tool
* recreate historical state to enable historical queries

* fixes

* load genesis from network metadata
* check checkpoint block root against state
* fix invalid block root in rest json decoding
* odds and ends

* retry looking for epoch-boundary checkpoint blocks
2022-01-17 10:27:08 +01:00
sacha df8d744a47
book updates (min viable changes prior to release) (#3283)
* min viable changes

* make clear keymanager api is not ready for mainnet

* Update docs/the_nimbus_book/src/keymanager-api.md

Co-authored-by: Jacek Sieka <jacek@status.im>

* Update docs/the_nimbus_book/src/keymanager-api.md

Co-authored-by: zah <zahary@gmail.com>

Co-authored-by: Jacek Sieka <jacek@status.im>
Co-authored-by: zah <zahary@gmail.com>
2022-01-14 13:37:21 +01:00
Zahary Karadjov e67773a495
Document the Keymanager API in the book 2022-01-13 20:19:07 +02:00
Jacek Sieka 6f7e0e3393
REST cleanups (#3255)
* REST cleanups

* reject out-of-range committee requests
* print all hex values as lower-case
* allow requesting state information by head state root
* turn `DomainType` into array (follow spec)
* `uint_to_bytesXX` -> `uint_to_bytes` (follow spec)
* fix wrong dependent root in `/eth/v1/validator/duties/proposer/`
* update documentation - `--subscribe-all-subnets` is no longer needed
when using the REST interface with validator clients
* more fixes
* common helpers for dependent block
* remove test rules obsoleted by more strict epoch tests
* fix trailing commas

* Update docs/the_nimbus_book/src/rest-api.md
* Update docs/the_nimbus_book/src/rest-api.md

Co-authored-by: sacha <sacha@status.im>
2022-01-08 22:06:34 +02:00
Jacek Sieka ba99c8fe4f
update era file documentation / impl (#3226)
Overhaul of era files, including documentation and reference
implementations

* store blocks, then state, then slot indices for easy lookup at low
cost
* document era file rationale
* altair+ support in era writer
2022-01-07 11:13:19 +01:00
zah fba1f08a5e
Implement #3129 (Optimized history traversals in the REST API) (#3219)
* Fix REST some rest call signatures and implement a simple API benchmark tool

* Implement #3129 (Optimized history traversals in the REST API)

Other notable changes:

The `updateStateData` procedure in the `blockchain_dag.nim` module is
optimized to not rewind down to the last snapshot state saved in the
database if the supplied input state can be used as a starting point
instead.

* Disallow await in withStateForBlockSlot
2022-01-05 15:49:10 +01:00
Zahary Karadjov 54d0d588b1 Implementation of the Keymanager API (BETA)
https://github.com/ethereum/keymanager-APIs
2022-01-04 18:51:45 +02:00
tersec da017d2ca5
update from phase0/altair v1.1.6 URLs to v1.1.8 spec URLs (#3238) 2022-01-04 03:57:15 +00:00
Jacek Sieka c270ec21e4
Validator monitoring (#2925)
Validator monitoring based on and mostly compatible with the
implementation in Lighthouse - tracks additional logs and metrics for
specified validators so as to stay on top on performance.

The implementation works more or less the following way:
* Validator pubkeys are singled out for monitoring - these can be
running on the node or not
* For every action that the validator takes, we record steps in the
process such as messages being seen on the network or published in the
API
* When the dust settles at the end of an epoch, we report the
information from one epoch before that, which coincides with the
balances being updated - this is a tradeoff between being correct
(waiting for finalization) and providing relevant information in a
timely manner)
2021-12-20 20:20:31 +01:00
Jacek Sieka 03005f48e1
Backfill support for ChainDAG (#3171)
In the ChainDAG, 3 block pointers are kept: genesis, tail and head. This
PR adds one more block pointer: the backfill block which represents the
block that has been backfilled so far.

When doing a checkpoint sync, a random block is given as starting point
- this is the tail block, and we require that the tail block has a
corresponding state.

When backfilling, we end up with blocks without corresponding states,
hence we cannot use `tail` as a backfill pointer - there is no state.

Nonetheless, we need to keep track of where we are in the backfill
process between restarts, such that we can answer GetBeaconBlocksByRange
requests.

This PR adds the basic support for backfill handling - it needs to be
integrated with backfill sync, and the REST API needs to be adjusted to
take advantage of the new backfilled blocks when responding to certain
requests.

Future work will also enable moving the tail in either direction:
* pruning means moving the tail forward in time and removing states
* backwards means recreating past states from genesis, such that
intermediate states are recreated step by step all the way to the tail -
at that point, tail, genesis and backfill will match up.
* backfilling is done when backfill != genesis - later, this will be the
WSS checkpoint instead
2021-12-13 14:36:06 +01:00
Eugene Kabanov b05734f610
Backward sync support for SyncManager. (#3131)
* Unbundle SyncQueue from sync_manager.nim.
Unbundle Peer scores constants to peer_scores.nim.
Add Forward/Backward enum.

* Further improvements and tests.

* Adopt getRewindPoint() and fix MissingParent handler.

* Remove unused procedures.
Refactor `result` usage.
Fix resetWait().

* Add all the tests and fix the issue with rewind point.

* Fix get() issue.

* Fix flaky tests.

* test fixes

Co-authored-by: Jacek Sieka <jacek@status.im>
2021-12-08 22:15:29 +01:00
Jacek Sieka 1a8b7469e3
move quarantine outside of chaindag (#3124)
* move quarantine outside of chaindag

The quarantine has been part of the ChainDAG for the longest time, but
this design has a few issues:

* the function in which blocks are verified and added to the dag becomes
reentrant and therefore difficult to reason about - we're currently
using a stateful flag to work around it
* quarantined blocks bypass the processing queue leading to a processing
stampede
* the quarantine flow is unsuitable for orphaned attestations - these
should also should be quarantined eventually

Instead of processing the quarantine inside ChainDAG, this PR moves
re-queueing to `block_processor` which already is responsible for
dealing with follow-up work when a block is added to the dag

This sets the stage for keeping attestations in the quarantine as well.

Also:

* make `BlockError` `{.pure.}`
* avoid use of `ValidationResult` in block clearance (that's for gossip)
2021-12-06 10:49:01 +01:00
tersec e6921f808f
cleanups, partly from kintsugi branch (#3161)
* cleanups, partly from kintsugi branch

* re-export shortLog(EthBlock) and preserve exception messages in batchVerify and processBatch
2021-12-05 17:32:41 +00:00
tersec cc51f3fd12
v1.1.{5 -> 6} phase 0 and altair spec URL updates (#3157) 2021-12-03 17:40:23 +00:00
sacha a1db321f0a
Update eth1.md
remover warning
2021-12-01 15:23:51 +01:00
tersec 8da2f5307d
remove obsolete amphora interop documentation (#3145) 2021-12-01 14:01:24 +01:00
sacha 335f7c1072
Update infura-guide.md
revert endpoint advice
2021-11-30 18:37:54 +01:00
rasberrylasagna 13280f230c
[FEAT] Updating some infura doc (#3137)
* [FEAT] updating some infura doc

* [FIX] Suggested changes

Co-authored-by: sacha <sacha.saint-leger@ethereum.org>

Co-authored-by: sacha <sacha.saint-leger@ethereum.org>
2021-11-30 12:54:14 +01:00
tersec cc0dbd5bc0
implement terminal-total-difficulty-override; keep kintsugi TTDs consistent (#3118)
* implement terminal-total-difficulty-override; keep TTD consistent for m2 scripts/docs

* use Option[uint64] instead of uint64
2021-11-25 10:53:31 +00:00
tersec 9e395011d9
update 22 spec URLs to v1.1.5 (#3111) 2021-11-18 08:08:00 +00:00
tersec 39f6a6534e
document how to run merge local testnet with Nethermind (#3110) 2021-11-17 20:45:39 +01:00
Ștefan Talpalaru c673525591
book: add FRITZ!Box UPnP setup (#3106)
* book: add FRITZ!Box UPnP setup

* Update docs/the_nimbus_book/src/networking.md

* Update docs/the_nimbus_book/src/networking.md

* Update docs/the_nimbus_book/src/networking.md

Co-authored-by: sacha <sacha@status.im>
2021-11-16 10:46:27 +00:00
sacha 14cb68c72f
Book: upgrade / downgrade page (#3105)
* cp

* cp
2021-11-15 20:07:30 +01:00
tersec c5fa7387f9
update kintsugi docs (#3104) 2021-11-15 17:01:27 +00:00
Jacek Sieka ec650c7fd7
Support starting from altair (#3054)
* Support starting from altair

* hide `finalized-checkpoint-` - they are incomplete and usage may cause
crashes
* remove genesis detection code (broken, obsolete)
* enable starting ChainDAG from altair checkpoints - this is a
prerequisite for checkpoint sync (TODO: backfill)
* tighten checkpoint state conditions
* show error when starting from checkpoint with existing database (not
supported)
* print rest-compatible JSON in ncli/state_sim
* altair/merge support in ncli
* more altair/merge support in ncli_db
* pre-load header to speed up loading
* fix forked block decoding
2021-11-10 13:39:08 +02:00
zah 2c0be052a5
Reduce the logging-related breaking changes (#3070)
* Reduce the logging-related breaking changes

* Don't disable the stdout log when the `--log-file` option is used
* Rename `--log-stdout` to `--log-format` and hide it
* Deprecate the `--log-file` option
2021-11-10 11:02:18 +02:00
sacha 9d82606eba
Update logging.md 2021-11-09 15:38:48 +01:00
sacha db612ae906
Update logging.md
add warning
2021-11-09 15:38:32 +01:00
sacha 87fa67045f
Update logging.md
replace `--log-stdout` with -`-log-format`. remove references to `--log-file`
2021-11-09 15:21:48 +01:00
tersec 48eba59971
manually-verified v1.1.5 spec URL updates (#3068) 2021-11-09 08:54:59 +00:00
tersec 2e868dc2ba
mass/mechanical update of 1.1.4 phase0 and altair spec URLs to 1.1.5 (#3067) 2021-11-09 07:40:41 +00:00
sacha 624345bcee
Update logging.md
mainly style edits
2021-11-08 22:05:11 +01:00
tersec 2c8600e746
mass/mechanical update of 1.1.3 phase0 spec URLs to 1.1.4 in markdown (#3059) 2021-11-08 09:26:18 +00:00
Jacek Sieka ea0a191723
Better REST/RPC error messages (#3046)
* Better REST/RPC error messages
* homogenise block logging (root first)
* homegenise message verification pipeline (verify in
`gossip_verification`, act in `eth2_processor`)
* use `subcommitteeIdx` consistently
* log each sent contribution
* fix block_sim
* fix block topic
* don't recalc root on gossip block validation
* move position loop into sync pool
2021-11-05 17:39:47 +02:00
Ștefan Talpalaru b09cd71348 build: show stderr by default
Also added: `-d:cwarnings` and `-d:limitStackUsage`
2021-11-05 16:06:48 +02:00
sacha 122055b65e
cp (#3053) 2021-11-05 11:33:28 +01:00
Jacek Sieka 233d756518
Logging and startup improvements (#3038)
* Logging and startup improvements

Color support for released binaries!

* startup scripts no longer log to file by default - this only affects
source builds - released binaries don't support file logging
* add --log-stdout option to control logging to stdout (colors, json)
* detect tty:s vs redirected logs and log accordingly
* add option to disable log colors at runtime
* simplify several "common" logs, showing the most important information
earlier and more clearly
* remove line numbers / file information / tid - these take up space and
are of little use to end users
  * still enabled in debug builds and tools
* remove `testnet_servers_image` compile-time option
* server images, released binaries and compile-from-source now offer
the same behaviour and features
* fixes https://github.com/status-im/nimbus-eth2/issues/2326
* fixes https://github.com/status-im/nimbus-eth2/issues/1794
* remove instanteneous block speed from sync message, keeping only
average

before:

```
INF 2021-10-28 16:45:59.000+02:00 Slot start                                 topics="beacnde" tid=386429 file=nimbus_beacon_node.nim:884 lastSlot=2384027 wallSlot=2384028 delay=461us84ns peers=0 head=75a10ee5:3348 headEpoch=104 finalized=cd6804ba:3264 finalizedEpoch=102 sync="wwwwwwwwww:0:0.0000:0.0000:00h00m (3348)"
INF 2021-10-28 16:45:59.046+02:00 Slot end                                   topics="beacnde" tid=386429 file=nimbus_beacon_node.nim:821 slot=2384028 nextSlot=2384029 head=75a10ee5:3348 headEpoch=104 finalizedHead=cd6804ba:3264 finalizedEpoch=102 nextAttestationSlot=-1 nextProposalSlot=-1 nextActionWait=n/a
```

after:

```
INF 2021-10-28 22:43:23.033+02:00 Slot start                                 topics="beacnde" slot=2385815 epoch=74556 sync="DDPDDPUDDD:10:5.2258:01h19m (2361088)" peers=37 head=eacd2dae:2361096 finalized=73782:a4751487 delay=33ms687us715ns
INF 2021-10-28 22:43:23.291+02:00 Slot end                                   topics="beacnde" slot=2385815 nextActionWait=n/a nextAttestationSlot=-1 nextProposalSlot=-1 head=eacd2dae:2361096
```

* fix comment

* documentation updates

* mention `--log-file` may be deprecated in the future
* update various docs
2021-11-02 18:06:36 +01:00
sacha 56c6d6b583
update troubleshooting (#3044)
* update troubleshooting

* remove pyrmont references
2021-11-01 17:46:59 +01:00
tersec 36e37bda40
v1.1.3 spec refs URLs (#3036) 2021-10-27 18:40:17 +00:00
tersec 968a5b3bec
some spec refs to v1.1.3 (#3032) 2021-10-26 18:42:48 +00:00
tersec 8307e9c601
mechanical non-merge v1.1.2 to v1.1.3 spec URL updates (#3030) 2021-10-26 16:44:23 +00:00
sacha 07e4dd5f23
edit networking, remove references to pyrmont (#3027) 2021-10-25 19:40:29 +02:00
sacha dc3c23788a
update networking, update hardware requirements, set up health.md (#3020) 2021-10-21 16:18:46 +02:00
sacha c3c8e27011
Network setup update (#2999)
* cp

* cp

* Update docs/the_nimbus_book/src/faq.md

Co-authored-by: Ștefan Talpalaru <stefantalpalaru@yahoo.com>

* Update docs/the_nimbus_book/src/faq.md

Co-authored-by: Ștefan Talpalaru <stefantalpalaru@yahoo.com>

* Update docs/the_nimbus_book/src/networking.md

Co-authored-by: Jacek Sieka <jacek@status.im>

* Update docs/the_nimbus_book/src/networking.md

Co-authored-by: Jacek Sieka <jacek@status.im>

* Update docs/the_nimbus_book/src/networking.md

Co-authored-by: Jacek Sieka <jacek@status.im>

* Update docs/the_nimbus_book/src/networking.md

Co-authored-by: Jacek Sieka <jacek@status.im>

* Update docs/the_nimbus_book/src/networking.md

Co-authored-by: Jacek Sieka <jacek@status.im>

* fix links

* new line

* extip edits

* update read logs

* cp

* cp

Co-authored-by: Ștefan Talpalaru <stefantalpalaru@yahoo.com>
Co-authored-by: Jacek Sieka <jacek@status.im>
2021-10-21 15:34:46 +02:00
Jacek Sieka 421bf936ff
odds and ends (#3015)
* `allSyncCommittees` => `allSyncSubcommittees`
* simplify `_snappy` topic generation (avoid pointless string copies)
* simplify gossip id generator (avoid pointless string copies)
* avoid redundant syncnet ENR updates
* simplify topic validation (allow only validated topics)
2021-10-21 15:09:19 +02:00
tersec 6cc8757930
update 18 spec URLs to v1.1.2; switch from eth2.0-specs to consensus-specs (#2990) 2021-10-14 06:30:21 +00:00
tersec 2ad1b7366a
update 62 spec URLs to v1.1.2 (#2979) 2021-10-12 10:17:37 +00:00
tersec 0ae736f397
update 67 spec URLs to v1.1.2 (#2977) 2021-10-12 08:09:59 +00:00
tersec bbd51c02ec
replace obsolete Rayonism interop docs with Amphora interop docs (#2974) 2021-10-11 07:06:13 +00:00
sacha 6f1cfc2a8c
add testing apis (#2953) 2021-10-04 18:34:24 +02:00
zah 653fc72b2e
Document that the REST API is out of beta (#2947) 2021-10-04 13:55:02 +02:00
tersec ca4c6b4c5c
update 30 spec URLs to v1.1.0 (#2914) 2021-09-28 14:01:46 +00:00
tersec aec5cf2b1b
update 31 spec reference URLs to v1.1.0 (#2910) 2021-09-28 07:46:06 +00:00
Etan Kissling cc30bf63b4
update gossip_processing and attestation docs (#2860)
The README file explaining gossip_processing, and the attestation_flow
docs were no longer accurate, as attestations and aggregates no longer
go through a queue (pending batching). This patch updates the docs
accordingly. It also improves some grammar and fixes some typos.
2021-09-27 15:11:10 +02:00
sacha 2ec34e6761
Book edits (#2887)
* cp

* cp
2021-09-22 15:52:42 +02:00
Etan Kissling 7b6976d3fd
bump Nim version in developer docs (#2841)
The developer documentation contains a comment mentioning the version of
Nim that the project is tested against. This comment was not updated for
a while (last Nim update in #2508). Bumping now to be in sync once more.
2021-09-01 14:55:54 +02:00
tersec 9725d15a3e
update spec references from eth2.0-specs to consensus-specs and to v1.1.0-beta.2 (#2822) 2021-08-26 10:21:52 +02:00
Zachinquarantine 860897172a
connect-eth2.md: fix small typo (#2804) 2021-08-22 11:24:56 +02:00
Jacek Sieka c3a0e4ec16
API doc fixes
`getSpecPreset` was removed pre-1.0.0 and has never worked - debug calls
were renamed around 1.1.0.
2021-08-16 18:41:13 +02:00
0xmiel 01f78fb93e
update links to joe's guides (#2775) 2021-08-11 15:29:49 +02:00
0xmiel fe96c3e993
add page on security audit (#2774)
* add page on security audit

* incorporate feedback
2021-08-11 15:16:02 +02:00
Jacek Sieka 3d7bee8502
REST API client, JSON-RPC cleanups (#2756)
This refactoring puts the JSON-RPC and REST APIs on more equal footing
by renaming and moving things around, creating a separation between
client and server, and documenting what they are - the aim is to have a
simple-to-use base to start from when developing API clients, as well as
make it easier to navigate the code when looking for the legacy JSON-RPC
interface vs the new REST API.

* move REST client, serialization and supporting types to spec/eth2_apis
* REST stuff now starts with `rest_`, JSON-RPC stuff starts with `rpc_`,
more or less
* simplify imports such that there's a simple module to import for both
server and client
* map REST type and proc names to yaml spec more closely - in
particular, reuse operation and type names in `rest_types` to make
comparisons against spec more easy
* cleaner separation between client and server modules - modules common
between server and client such as `rest_types` and serialization move to
the spec folder - this allows the client to be built with less knowledge
about server internals
2021-08-03 17:17:11 +02:00