Nim implementation of the Ethereum Beacon Chain https://nimbus.guide
Go to file
Eugene Kabanov 3b6f4fab4a
New validator client using REST API. (#2651)
* Initial commit.

* Exporting getConfig().

* Add beacon node checking procedures.

* Post rebase fixes.

* Use runSlotLoop() from nimbus_beacon_node.
Fallback implementation.
Fixes for ETH2 REST serialization.

* Add beacon_clock.durationToNextSlot().
Move type declarations from beacon_rest_api to json_rest_serialization.
Fix seq[ValidatorIndex] serialization.
Refactor ValidatorPool and add some utility procedures.
Create separate version of validator_client.

* Post-rebase fixes.
Remove CookedPubKey from validator_pool.nim.

* Now we should be able to produce attestations and aggregate and proofs.
But its not working yet.

* Debugging attestation sending.

* Add durationToNextAttestation.
Optimize some debug logs.
Fix aggregation_bits encoding.
Bump chronos/presto.

* Its alive.

* Fixes for launch_local_testnet script.
Bump chronos.

* Switch client API to not use `/api` prefix.

* Post-rebase adjustments.

* Fix endpoint for publishBlock().

* Add CONFIG_NAME.
Add more checks to ensure that beacon_node is compatible.

* Add beacon committee subscription support to validator_client.

* Fix stacktrace should be an array of strings.
Fix committee subscriptions should not be `data` keyed.

* Log duration to next block proposal.

* Fix beacon_node_status import.

* Use jsonMsgResponse() instead of jsonError().

* Fix graffityBytes usage.
Remove unnecessary `await`.
Adjust creation of SignedBlock instance.
Remove legacy files.

* Rework durationToNextSlot() and durationToNextEpoch() to use `fromNow`.

* Fix race condition for block proposal and attestations for same slot.
Fix local_testnet script to properly kill tasks on Windows.
Bump chronos and nim-http-tools, to allow connections to infura.io (basic auth).

* Catch services errors.
Improve performance of local_testnet.sh script on Windows.
Fix race condition when attestation producing.

* Post-rebase fixes.

* Bump chronos and presto.

* Calculate block publishing delay.
Fix pkill in one more place.

* Add error handling and timeouts to firstSuccess() template.
Add onceToAll() template.
Add checkNodes() procedure.
Refactor firstSuccess() template.
Add error checking to api.nim calls.

* Deprecated usage onceToAll() for better stability.
Address comment and send attestations asap.

* Avoid unnecessary loop when calculating minimal duration.
2021-07-13 13:15:07 +02:00
.github Merge branch 'stable' into testing 2021-06-21 15:52:51 +02:00
.vscode vscode: fix tasks 2019-11-25 13:51:47 +01:00
beacon_chain New validator client using REST API. (#2651) 2021-07-13 13:15:07 +02:00
benchmarks openarray -> openArray 2020-11-03 23:23:10 +02:00
docker bin dist related docs (#2684) 2021-07-05 12:31:25 +02:00
docs Implement the new Altair req/resp protocols (#2676) 2021-07-07 12:09:47 +03:00
grafana Prometheus/Grafana changes (#2092) 2020-12-02 21:47:00 +01:00
media forgot the image 2020-06-11 16:00:56 +02:00
multinet Remove multinet scripts and forward to their new location [skip ci] 2019-10-29 19:18:01 +02:00
nbench Implement split preset/config support (#2710) 2021-07-12 15:01:38 +02:00
ncli Implement split preset/config support (#2710) 2021-07-12 15:01:38 +02:00
nfuzz Implement split preset/config support (#2710) 2021-07-12 15:01:38 +02:00
research Implement split preset/config support (#2710) 2021-07-12 15:01:38 +02:00
scripts New validator client using REST API. (#2651) 2021-07-13 13:15:07 +02:00
tests re-add minimal tests, now that const sanity checks gone (#2715) 2021-07-13 08:38:35 +02:00
tools Windows binary release 2021-02-04 10:25:44 +02:00
vendor New validator client using REST API. (#2651) 2021-07-13 13:15:07 +02:00
wasm scripts: make shebang more portable (#2216) 2021-01-08 11:29:43 +01:00
.appveyor.yml clean up dependencies (#2008) 2020-11-13 16:00:45 +01:00
.editorconfig Makefile targets for ctail and ntu 2020-06-22 17:30:04 +03:00
.gitignore Consolidate modules by areas [part 1] (#2365) 2021-03-02 11:27:45 +01:00
.gitmodules Fix sync_manager.nim to return proper status. 2021-04-09 21:42:13 +03:00
.nvmrc Local sim now includes a prometheus server, a Ganache instance and a simulation dashboard server (in Tmux mode) 2020-06-22 17:30:04 +03:00
.travis.yml Try to resurrect Travis ARM64 and only use Travis for ARM test (#2167) 2020-12-09 22:53:28 +01:00
AllTests-mainnet.md Implement split preset/config support (#2710) 2021-07-12 15:01:38 +02:00
CHANGELOG.md v1.4.0 2021-06-21 14:02:12 +02:00
FixtureAll-mainnet.md update spec test fixtures to v1.1.0-alpha.8 (#2687) 2021-06-29 16:47:09 +00:00
FixtureSSZConsensus-mainnet.md update spec test fixtures to v1.1.0-alpha.8 (#2687) 2021-06-29 16:47:09 +00:00
FixtureSSZConsensus-minimal.md re-add minimal tests, now that const sanity checks gone (#2715) 2021-07-13 08:38:35 +02:00
FixtureSSZGeneric-mainnet.md Reform the networking layer in order to handle the new stricter SSZ API 2020-05-24 19:00:34 +03:00
Jenkinsfile Store REST test logs as CI artifacts. (#2699) 2021-07-06 08:40:18 +00:00
Jenkinsfile.benchmarks Jenkins: don't abort older jobs in master/devel (#1746) 2020-09-25 10:26:59 +00:00
LICENSE-APACHEv2 initial commit 2018-07-20 15:46:03 +02:00
LICENSE-MIT initial commit 2018-07-20 15:46:03 +02:00
Makefile re-add minimal tests, now that const sanity checks gone (#2715) 2021-07-13 08:38:35 +02:00
README.md Update README links 2021-04-29 10:46:09 +02:00
azure-pipelines.yml clean up dependencies (#2008) 2020-11-13 16:00:45 +01:00
config.nims Merge stable into unstable 2021-05-20 13:50:40 +03:00
env.sh REST API test framework and tests. (#2492) 2021-04-27 23:46:24 +03:00
run-mainnet-beacon-node.sh Windows binary release 2021-02-04 10:25:44 +02:00
run-prater-beacon-node.sh Support for the prater testnet 2021-03-15 21:51:24 +02:00
run-pyrmont-beacon-node.sh Windows binary release 2021-02-04 10:25:44 +02:00

README.md

Nimbus Eth2 (Beacon Chain)

Github Actions CI License: Apache License: MIT Stability: experimental

Discord: Nimbus Status: #nimbus-general

Nimbus-eth2 is a extremely efficient Beacon Chain client for participating in the Ethereum Proof of Stake protocol. It performs well on embedded systems, resource-restricted devices -- including Raspberry Pis and mobile devices -- the low resource usage also makes it an excellet choice for running together with an Ethereum client on a server or a desktop where it simply takes up less resources.

Documentation

You can find the information you need to run a beacon node and operate as a validator in The Book.

The Quickstart in particular will help you get connected to the Pyrmont Testnet and eth2 Mainnet quickly!

You can check where the beacon chain fits in the Ethereum ecosystem our Two-Point-Oh series: https://our.status.im/tag/two-point-oh/

Donations

If you'd like to contribute to Nimbus development, our donation address is 0x70E47C843E0F6ab0991A3189c28F2957eb6d3842

Branch guide

  • stable - latest stable release - this branch is recommended for most users
  • testing - pre-release branch with features and bugfixes slated for the next stable release - this branch is suitable for use on testnets and for adventerous users that want to live on the edge.
  • unstable - main development branch against which PR's are merged - if you want to contribute to Nimbus, start here.

Developer resources

To build tools that interact with Nimbus while it's running, we expose an RPC API.

To get started with developing Nimbus itself, see the developer handbook. The code follows the Status Nim Style Guide.

Nimbus is built in the Nim language - the compiler is automatically installed when building the project for the first time. More information - in particular security-related information about the language - can be found in the Auditor Handbook.

Tooling and utilities

We provide several tools to interact with ETH2 and the data in the beacon chain:

  • ncli - command line tool with pretty printers, SSZ decoders, state transition helpers to interact with Eth2 data structures and functions
  • ncli_db - command line tool to perform surgery on the Nimbus sqlite database
  • inspector - command line tool for interacting with the peer to peer protocols in eth2
  • multinet - a set of scripts to build and run several Eth2 clients locally

For researchers

State transition simulation

The state transition simulator can quickly run the Beacon chain state transition function in isolation and output JSON snapshots of the state. The simulation runs without networking and blocks are processed without slot time delays.

# build and run the state simulator, then display its help ("-d:release" speeds it
# up substantially, allowing the simulation of longer runs in reasonable time)
make NIMFLAGS="-d:release" state_sim
build/state_sim --help

Local network simulation

The local network simulation will create a full peer-to-peer network of beacon nodes and validators on a single machine, and run the beacon chain in real time.

Parameters such as shard, validator counts, and data folders are configured vars.sh. They can be set in as environment variables before launching the simulation.

# Clear data files from your last run and start the simulation with a new genesis block:
make VALIDATORS=192 NODES=6 USER_NODES=1 eth2_network_simulation

# In another terminal, get a shell with the right environment variables set:
./env.sh bash

# In the above example, the network is prepared for 7 beacon nodes but one of
# them is not started by default (`USER_NODES`) - this is useful to test
# catching up to the consensus. The following command will start the missing node.
./tests/simulation/run_node.sh 0 # (or the index (0-based) of the missing node)

# Running a separate node allows you to test sync as well as see what the action
# looks like from a single nodes' perspective.

By default, validators will be split in half between beacon node and validator client processes (50/50), communicating through the official validator API (for example with 192 validators and 6 nodes you will roughly end up with 6 beacon node and 6 validator client processes, where each of them will handle 16 validators), but if you don't want to use external validator clients and instead want to have all the validators handled by the beacon nodes you may use BN_VC_VALIDATOR_SPLIT=no as an additional argument to make eth2_network_simulation.

By default, the simulation will start from a pre-generated genesis state. If you wish to simulate the bootstrap process with a Ethereum 1.0 validator deposit contract, start the simulation with WAIT_GENESIS=yes

make eth2_network_simulation WAIT_GENESIS=yes

You can also separate the output from each beacon node in its own panel, using multitail:

make eth2_network_simulation USE_MULTITAIL="yes"

You can find out more about it in the development update.

Alternatively, fire up our experimental Vagrant instance with Nim pre-installed and give us yout feedback about the process!

Visualising simulation metrics

The generic instructions from the Nimbus repo apply here as well.

Specific steps:

# This will generate the Prometheus config on the fly, based on the number of
# nodes (which you can control by passing something like NODES=6 to `make`).
make VALIDATORS=192 NODES=6 USER_NODES=0 eth2_network_simulation

# In another terminal tab, after the sim started:
cd tests/simulation/prometheus
prometheus

The dashboard you need to import in Grafana is "grafana/beacon_nodes_Grafana_dashboard.json".

monitoring dashboard

Network inspection

The inspector tool can help monitor the libp2p network and the various channels where blocks and attestations are being transmitted, showing message and connectivity metadata. By default, it will monitor all ethereum 2 gossip traffic.

. ./env.sh
# Build inspector for minimal config:
./env.sh nim c -d:const_preset=minimal -o:build/inspector_minimal ncli/inspector.nim

# Build inspector for mainnet config:
./env.sh nim c -d:const_preset=mainnet -o:build/inspector_mainnet ncli/inspector.nim

# See available options
build/inspector_minimal --help

# Connect to a network from eth2 testnet repo bootstrap file - --decode option attempts to decode the messages as well
build/inspector_minimal --decode -b:$(curl -s https://raw.githubusercontent.com/eth2-clients/eth2-testnets/master/nimbus/testnet0/bootstrap_nodes.txt | head -n1)

CI setup

Local testnets run for 4 epochs each, to test finalization. That happens only on Jenkins Linux hosts, and their logs are available for download as artifacts, from the job's page. Don't expect these artifacts to be kept more than a day after the corresponding branch is deleted.

Jenkins artifacts

License

Licensed and distributed under either of

or

at your option. These files may not be copied, modified, or distributed except according to those terms.