Nim implementation of the Ethereum Beacon Chain https://nimbus.guide
Go to file
Jacek Sieka 61342c2449
limit by-root requests to non-finalized blocks (#3293)
* limit by-root requests to non-finalized blocks

Presently, we keep a mapping from block root to `BlockRef` in memory -
this has simplified reasoning about the dag, but is not sustainable with
the chain growing.

We can distinguish between two cases where by-root access is useful:

* unfinalized blocks - this is where the beacon chain is operating
generally, by validating incoming data as interesting for future fork
choice decisions - bounded by the length of the unfinalized period
* finalized blocks - historical access in the REST API etc - no bounds,
really

In this PR, we limit the by-root block index to the first use case:
finalized chain data can more efficiently be addressed by slot number.

Future work includes:

* limiting the `BlockRef` horizon in general - each instance is 40
bytes+overhead which adds up - this needs further refactoring to deal
with the tail vs state problem
* persisting the finalized slot-to-hash index - this one also keeps
growing unbounded (albeit slowly)

Anyway, this PR easily shaves ~128mb of memory usage at the time of
writing.

* No longer honor `BeaconBlocksByRoot` requests outside of the
non-finalized period - previously, Nimbus would generously return any
block through this libp2p request - per the spec, finalized blocks
should be fetched via `BeaconBlocksByRange` instead.
* return `Opt[BlockRef]` instead of `nil` when blocks can't be found -
this becomes a lot more common now and thus deserves more attention
* `dag.blocks` -> `dag.forkBlocks` - this index only carries unfinalized
blocks from now - `finalizedBlocks` covers the other `BlockRef`
instances
* in backfill, verify that the last backfilled block leads back to
genesis, or panic
* add backfill timings to log
* fix missing check that `BlockRef` block can be fetched with
`getForkedBlock` reliably
* shortcut doppelganger check when feature is not enabled
* in REST/JSON-RPC, fetch blocks without involving `BlockRef`

* fix dag.blocks ref
2022-01-21 13:33:16 +02:00
.github cron.yml: show Nim version 2021-12-15 20:43:17 +01:00
.vscode vscode: fix tasks 2019-11-25 13:51:47 +01:00
beacon_chain limit by-root requests to non-finalized blocks (#3293) 2022-01-21 13:33:16 +02:00
benchmarks Implement #3129 (Optimized history traversals in the REST API) (#3219) 2022-01-05 15:49:10 +01:00
docker dist-macos-arm64: use -mcpu=apple-a13 (#3202) 2021-12-15 23:08:42 +01:00
docs Update start-syncing.md (#3300) 2022-01-20 17:15:09 +01:00
grafana Grafana: don't interpolate over null values (#3173) 2021-12-08 14:21:48 +01:00
media forgot the image 2020-06-11 16:00:56 +02:00
ncli limit by-root requests to non-finalized blocks (#3293) 2022-01-21 13:33:16 +02:00
nfuzz move `state_transition` to `Result` (#3284) 2022-01-17 12:19:58 +01:00
research complete switch to beacon_chain/specs/datatypes/bellatrix (#3295) 2022-01-18 13:36:52 +00:00
scripts rename upgrade_to_{merge,bellatrix}; detect unchanging spec YAMLs (#3265) 2022-01-10 09:39:43 +00:00
tests limit by-root requests to non-finalized blocks (#3293) 2022-01-21 13:33:16 +02:00
tools Grafana: add 3 more panels (#2946) 2021-10-04 12:04:33 +02:00
vendor Slashing prot interchange tests v5.2.1 (#3277) 2022-01-20 17:14:06 +01:00
wasm scripts: make shebang more portable (#2216) 2021-01-08 11:29:43 +01:00
.appveyor.yml Logging and startup improvements (#3038) 2021-11-02 18:06:36 +01:00
.editorconfig Makefile targets for ctail and ntu 2020-06-22 17:30:04 +03:00
.gitignore time: spring cleaning (#3262) 2022-01-11 11:01:54 +01:00
.gitmodules updating eth2-networks (#3250) 2022-01-06 15:12:52 +01:00
.nvmrc Local sim now includes a prometheus server, a Ganache instance and a simulation dashboard server (in Tmux mode) 2020-06-22 17:30:04 +03:00
.travis.yml Logging and startup improvements (#3038) 2021-11-02 18:06:36 +01:00
AllTests-mainnet.md limit by-root requests to non-finalized blocks (#3293) 2022-01-21 13:33:16 +02:00
CHANGELOG.md Version 1.6.0 2022-01-14 13:52:06 +02:00
ConsensusSpecPreset-mainnet.md add Bellatrix fork and transition tests; "Ethereum Foundation" -> EF (#3242) 2022-01-05 09:42:56 +01:00
ConsensusSpecPreset-minimal.md add Bellatrix fork and transition tests; "Ethereum Foundation" -> EF (#3242) 2022-01-05 09:42:56 +01:00
Jenkinsfile increase Jenkins CI timeout to 4 hours (#3278) 2022-01-13 02:05:38 +00:00
Jenkinsfile.benchmarks Jenkins: don't abort older jobs in master/devel (#1746) 2020-09-25 10:26:59 +00:00
LICENSE-APACHEv2 initial commit 2018-07-20 15:46:03 +02:00
LICENSE-MIT initial commit 2018-07-20 15:46:03 +02:00
Makefile More efficient reward data persistance; Address review comments 2022-01-18 01:56:56 +02:00
README.md update 22 spec URLs to v1.1.5 (#3111) 2021-11-18 08:08:00 +00:00
azure-pipelines.yml Logging and startup improvements (#3038) 2021-11-02 18:06:36 +01:00
config.nims Apple M1: proper "-march=native" substitute 2021-12-17 13:27:53 +02:00
env.sh [skip ci] Allow the WEB3_URL to be specified in the user .env file 2021-10-19 18:01:03 +03:00
run-mainnet-beacon-node.sh /bin/bash -> /usr/bin/env bash (needed on NixOS) 2021-11-15 15:45:01 +02:00
run-prater-beacon-node.sh /bin/bash -> /usr/bin/env bash (needed on NixOS) 2021-11-15 15:45:01 +02:00
run-pyrmont-beacon-node.sh /bin/bash -> /usr/bin/env bash (needed on NixOS) 2021-11-15 15:45:01 +02:00

README.md

Nimbus Eth2 (Beacon Chain)

Github Actions CI License: Apache License: MIT Stability: experimental

Discord: Nimbus Status: #nimbus-general

Nimbus-eth2 is an extremely efficient consensus layer (eth2) client implementation. It performs well on embedded systems and resource-restricted devices -- including Raspberry Pis and mobile devices. It's low resource usage also makes it an excellent choice for running together with an Ethereum client on a server or a desktop (where it simply takes up fewer resources).

Documentation

You can find the information you need to run a beacon node and operate as a validator in The Book.

The Quickstart in particular will help you quickly connect to either mainnet or the Prater testnet.

Quickly test your tooling against Nimbus

The Nimbus REST api is now available from:

Note that right now these are very much unstable testing instances. They may be unresponsive at times - so please do not rely on them for validating. We may also disable them at any time.

You can check where the beacon chain fits in the Ethereum ecosystem in our Two-Point-Oh series: https://our.status.im/tag/two-point-oh/

Donations

If you'd like to contribute to Nimbus development, our donation address is 0x70E47C843E0F6ab0991A3189c28F2957eb6d3842

Branch guide

  • stable - latest stable release - this branch is recommended for most users
  • testing - pre-release branch with features and bugfixes slated for the next stable release - this branch is suitable for use on testnets and for adventerous users that want to live on the edge.
  • unstable - main development branch against which PR's are merged - if you want to contribute to Nimbus, start here.

Developer resources

To build tools that interact with Nimbus while it's running, we expose an RPC API.

To get started with developing Nimbus itself, see the developer handbook. The code follows the Status Nim Style Guide.

Nimbus is built in the Nim language - the compiler is automatically installed when building the project for the first time. More information - in particular security-related information about the language - can be found in the Auditor Handbook.

Tooling and utilities

We provide several tools to interact with ETH2 and the data in the beacon chain:

  • ncli - command line tool with pretty printers, SSZ decoders, state transition helpers to interact with Eth2 data structures and functions
  • ncli_db - command line tool to perform surgery on the Nimbus sqlite database
  • multinet - a set of scripts to build and run several Eth2 clients locally

For researchers

State transition simulation

The state transition simulator can quickly run the Beacon chain state transition function in isolation and output JSON snapshots of the state. The simulation runs without networking and blocks are processed without slot time delays.

# build and run the state simulator, then display its help ("-d:release" speeds it
# up substantially, allowing the simulation of longer runs in reasonable time)
make NIMFLAGS="-d:release" state_sim
build/state_sim --help

Local network simulation

The local network simulation will create a full peer-to-peer network of beacon nodes and validators on a single machine, and run the beacon chain in real time.

Parameters such as shard, validator counts, and data folders are configured vars.sh. They can be set in as environment variables before launching the simulation.

# Clear data files from your last run and start the simulation with a new genesis block:
make VALIDATORS=192 NODES=6 USER_NODES=1 eth2_network_simulation

# In another terminal, get a shell with the right environment variables set:
./env.sh bash

# In the above example, the network is prepared for 7 beacon nodes but one of
# them is not started by default (`USER_NODES`) - this is useful to test
# catching up to the consensus. The following command will start the missing node.
./tests/simulation/run_node.sh 0 # (or the index (0-based) of the missing node)

# Running a separate node allows you to test sync as well as see what the action
# looks like from a single nodes' perspective.

By default, validators will be split in half between beacon node and validator client processes (50/50), communicating through the common validator API (for example with 192 validators and 6 nodes you will roughly end up with 6 beacon node and 6 validator client processes, where each of them will handle 16 validators), but if you don't want to use external validator clients and instead want to have all the validators handled by the beacon nodes you may use BN_VC_VALIDATOR_SPLIT=no as an additional argument to make eth2_network_simulation.

By default, the simulation will start from a pre-generated genesis state. If you wish to simulate the bootstrap process with a Ethereum 1.0 validator deposit contract, start the simulation with WAIT_GENESIS=yes

make eth2_network_simulation WAIT_GENESIS=yes

You can also separate the output from each beacon node in its own panel, using multitail:

make eth2_network_simulation USE_MULTITAIL="yes"

You can find out more about it in the development update.

Alternatively, fire up our experimental Vagrant instance with Nim pre-installed and give us your feedback about the process!

Visualising simulation metrics

The generic instructions from the Nimbus repo apply here as well.

Specific steps:

# This will generate the Prometheus config on the fly, based on the number of
# nodes (which you can control by passing something like NODES=6 to `make`).
make VALIDATORS=192 NODES=6 USER_NODES=0 eth2_network_simulation

# In another terminal tab, after the sim started:
cd tests/simulation/prometheus
prometheus

The dashboard you need to import in Grafana is "grafana/beacon_nodes_Grafana_dashboard.json".

monitoring dashboard

CI setup

Local testnets run for 4 epochs each, to test finalization. That happens only on Jenkins Linux hosts, and their logs are available for download as artifacts, from the job's page. Don't expect these artifacts to be kept more than a day after the corresponding branch is deleted.

Jenkins artifacts

License

Licensed and distributed under either of

or

at your option. These files may not be copied, modified, or distributed except according to those terms.