4207b127f9
* era: load blocks and states Era files contain finalized history and can be thought of as an alternative source for block and state data that allows clients to avoid syncing this information from the P2P network - the P2P network is then used to "top up" the client with the most recent data. They can be freely shared in the community via whatever means (http, torrent, etc) and serve as a permanent cold store of consensus data (and, after the merge, execution data) for history buffs and bean counters alike. This PR gently introduces support for loading blocks and states in two cases: block requests from rest/p2p and frontfilling when doing checkpoint sync. The era files are used as a secondary source if the information is not found in the database - compared to the database, there are a few key differences: * the database stores the block indexed by block root while the era file indexes by slot - the former is used only in rest, while the latter is used both by p2p and rest. * when loading blocks from era files, the root is no longer trivially available - if it is needed, it must either be computed (slow) or cached (messy) - the good news is that for p2p requests, it is not needed * in era files, "framed" snappy encoding is used while in the database we store unframed snappy - for p2p2 requests, the latter requires recompression while the former could avoid it * front-filling is the process of using era files to replace backfilling - in theory this front-filling could happen from any block and front-fills with gaps could also be entertained, but our backfilling algorithm cannot take advantage of this because there's no (simple) way to tell it to "skip" a range. * front-filling, as implemented, is a bit slow (10s to load mainnet): we load the full BeaconState for every era to grab the roots of the blocks - it would be better to partially load the state - as such, it would also be good to be able to partially decompress snappy blobs * lookups from REST via root are served by first looking up a block summary in the database, then using the slot to load the block data from the era file - however, there needs to be an option to create the summary table from era files to fully support historical queries To test this, `ncli_db` has an era file exporter: the files it creates should be placed in an `era` folder next to `db` in the data directory. What's interesting in particular about this setup is that `db` remains as the source of truth for security purposes - it stores the latest synced head root which in turn determines where a node "starts" its consensus participation - the era directory however can be freely shared between nodes / people without any (significant) security implications, assuming the era files are consistent / not broken. There's lots of future improvements to be had: * we can drop the in-memory `BlockRef` index almost entirely - at this point, resident memory usage of Nimbus should drop to a cool 500-600 mb * we could serve era files via REST trivially: this would drop backfill times to whatever time it takes to download the files - unlike the current implementation that downloads block by block, downloading an era at a time almost entirely cuts out request overhead * we can "reasonably" recreate detailed state history from almost any point in time, turning an O(slot) process into O(1) effectively - we'll still need caches and indices to do this with sufficient efficiency for the rest api, but at least it cuts the whole process down to minutes instead of hours, for arbitrary points in time * CI: ignore failures with Nim-1.6 (temporary) * test fixes Co-authored-by: Ștefan Talpalaru <stefantalpalaru@yahoo.com>
Nimbus Eth2 (Beacon Chain)
Nimbus-eth2 is an extremely efficient consensus layer (eth2) client implementation. While it's optimised for embedded systems and resource-restricted devices -- including Raspberry Pis, it's low resource usage also makes it an excellent choice for any server or desktop (where it simply takes up fewer resources).
- Documentation
- Related projects
- Donations
- Branch guide
- Developer resources
- Tooling and utilities
- For researchers
- License
Documentation
You can find the information you need to run a beacon node and operate as a validator in The Book.
The Quickstart in particular will help you quickly connect to either mainnet or the Prater testnet.
Quickly test your tooling against Nimbus
The Nimbus REST api is now available from:
- http://testing.mainnet.beacon-api.nimbus.team/
- http://unstable.mainnet.beacon-api.nimbus.team/
- http://unstable.prater.beacon-api.nimbus.team/
Note that right now these are very much unstable testing instances. They may be unresponsive at times - so please do not rely on them for validating. We may also disable them at any time.
Migrate from another client
This guide will take you through the basics of how to migrate to Nimbus from another client. See here for advanced options.
Related projects
- status-im/nimbus-eth1: Nimbus for Ethereum 1
- ethereum/consensus-specs: Consensus specification that this project implements
You can check where the beacon chain fits in the Ethereum ecosystem in our Two-Point-Oh series: https://our.status.im/tag/two-point-oh/
Donations
If you'd like to contribute to Nimbus development, our donation address is 0x70E47C843E0F6ab0991A3189c28F2957eb6d3842
Branch guide
stable- latest stable release - this branch is recommended for most userstesting- pre-release branch with features and bugfixes slated for the next stable release - this branch is suitable for use on testnets and for adventerous users that want to live on the edge.unstable- main development branch against which PR's are merged - if you want to contribute to Nimbus, start here.
Developer resources
To build tools that interact with Nimbus while it's running, we expose an RPC API.
To get started with developing Nimbus itself, see the developer handbook. The code follows the Status Nim Style Guide.
Nimbus is built in the Nim language - the compiler is automatically installed when building the project for the first time. More information - in particular security-related information about the language - can be found in the Auditor Handbook.
Tooling and utilities
We provide several tools to interact with ETH2 and the data in the beacon chain:
- ncli - command line tool with pretty printers, SSZ decoders, state transition helpers to interact with Eth2 data structures and functions
- ncli_db - command line tool to perform surgery on the Nimbus sqlite database
- multinet - a set of scripts to build and run several Eth2 clients locally
For researchers
State transition simulation
The state transition simulator can quickly run the Beacon chain state transition function in isolation and output JSON snapshots of the state. The simulation runs without networking and blocks are processed without slot time delays.
# build and run the state simulator, then display its help ("-d:release" speeds it
# up substantially, allowing the simulation of longer runs in reasonable time)
make NIMFLAGS="-d:release" state_sim
build/state_sim --help
Local network simulation
The local network simulation will create a full peer-to-peer network of beacon nodes and validators on a single machine, and run the beacon chain in real time.
Parameters such as shard, validator counts, and data folders are configured vars.sh. They can be set in as environment variables before launching the simulation.
# Clear data files from your last run and start the simulation with a new genesis block:
make VALIDATORS=192 NODES=6 USER_NODES=1 eth2_network_simulation
# In another terminal, get a shell with the right environment variables set:
./env.sh bash
# In the above example, the network is prepared for 7 beacon nodes but one of
# them is not started by default (`USER_NODES`) - this is useful to test
# catching up to the consensus. The following command will start the missing node.
./tests/simulation/run_node.sh 0 # (or the index (0-based) of the missing node)
# Running a separate node allows you to test sync as well as see what the action
# looks like from a single nodes' perspective.
By default, validators will be split in half between beacon node and validator
client processes (50/50), communicating through the
common validator API
(for example with 192 validators and 6 nodes you will roughly end up with 6
beacon node and 6 validator client processes, where each of them will handle 16
validators), but if you don't want to use external validator clients and instead
want to have all the validators handled by the beacon nodes you may use
BN_VC_VALIDATOR_SPLIT=no as an additional argument to make eth2_network_simulation.
By default, the simulation will start from a pre-generated genesis state. If you wish to
simulate the bootstrap process with a Ethereum 1.0 validator deposit contract, start the
simulation with WAIT_GENESIS=yes
make eth2_network_simulation WAIT_GENESIS=yes
You can also separate the output from each beacon node in its own panel, using multitail:
make eth2_network_simulation USE_MULTITAIL="yes"
You can find out more about it in the development update.
Alternatively, fire up our experimental Vagrant instance with Nim pre-installed and give us your feedback about the process!
Visualising simulation metrics
The generic instructions from the Nimbus repo apply here as well.
Specific steps:
# This will generate the Prometheus config on the fly, based on the number of
# nodes (which you can control by passing something like NODES=6 to `make`).
make VALIDATORS=192 NODES=6 USER_NODES=0 eth2_network_simulation
# In another terminal tab, after the sim started:
cd tests/simulation/prometheus
prometheus
The dashboard you need to import in Grafana is "grafana/beacon_nodes_Grafana_dashboard.json".
CI setup
Local testnets run for 4 epochs each, to test finalization. That happens only on Jenkins Linux hosts, and their logs are available for download as artifacts, from the job's page. Don't expect these artifacts to be kept more than a day after the corresponding branch is deleted.
License
Licensed and distributed under either of
- MIT license: LICENSE-MIT or http://opensource.org/licenses/MIT
or
- Apache License, Version 2.0, (LICENSE-APACHEv2 or http://www.apache.org/licenses/LICENSE-2.0)
at your option. These files may not be copied, modified, or distributed except according to those terms.