Add more documentation to the fluffy book and reorganize it (#1856)

Moved pretty much everything from the readme.md file now to the
book and structured it better to be more accessible.
This commit is contained in:
Kim De Mey 2023-10-24 18:18:30 +02:00 committed by GitHub
parent 35ee171013
commit 6e59cccc5b
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
11 changed files with 360 additions and 165 deletions

View File

@ -9,13 +9,15 @@
[![Status: #nimbus-general](https://img.shields.io/badge/Status-nimbus--general-blue.svg)](https://join.status.im/nimbus-general)
## Introduction
This folder holds the development of the Nimbus client implementation supporting
the Portal Network: Fluffy. The Portal Network is a project still heavily in
research phase and fully in flux. This client is thus still highly experimental.
Current status of specifications can be found in the
[portal-network-specs repository](https://github.com/ethereum/portal-network-specs/blob/master/portal-network.md).
Fluffy is the Nimbus client implementation of the
[Portal network specifications](https://github.com/ethereum/portal-network-specs/blob/master/portal-network.md).
You can find all the information you need to run a Fluffy node in
[The Fluffy Book](https://status-im.github.io/nimbus-eth1).
The [quickstart page](https://status-im.github.io/nimbus-eth1/quick-start.html)
in particular will help you to get quickly connected to the Portal network.
## Development Updates
@ -25,163 +27,6 @@ Monthly development updates are shared
To keep up to date with changes and development progress, follow the
[Nimbus blog](https://our.status.im/tag/nimbus/).
## How to Build & Run
### Prerequisites
- GNU Make, Bash and the usual POSIX utilities. Git 2.9.4 or newer.
### Build the Fluffy client
```bash
git clone git@github.com:status-im/nimbus-eth1.git
cd nimbus-eth1
make fluffy
# See available command line options
./build/fluffy --help
# Example command: Run the client and connect to a bootstrap node.
./build/fluffy --bootstrap-node:enr:<base64 encoding of ENR>
```
### Update and rebuild the Fluffy client
```bash
# From the nimbus-eth1 repository
git pull
# To bring the git submodules up to date
make update
make fluffy
```
### Run a Fluffy client on the public testnet
```bash
# Connect to the Portal testnet bootstrap nodes and enable the JSON-RPC APIs
./build/fluffy --rpc --table-ip-limit:1024 --bucket-ip-limit:24
```
The `table-ip-limit` and `bucket-ip-limit` options are needed to allow more
nodes with the same IPs in the routing tables. The default limits are there
as security measure. It is currently needed to increase the limits for the testnet
because the fleet of Fluffy nodes runs on only 2 machines / network interfaces.
There is a public [Portal testnet](https://github.com/ethereum/portal-network-specs/blob/master/testnet.md#portal-network-testnet)
which contains nodes of different clients.
Fluffy will default join the network through these bootstrap nodes.
You can also explicitly provide the `--network:testnet0` option to join this
network, or `--network:none` to not connect to any of these bootstrapn odes.
> **_Note:_** The `--network` option selects automatically a static set of
specific bootstrap nodes belonging to a "testnet". Currently `testnet0` is the
only option, which results in connecting to the
[testnet bootstrap nodes](https://github.com/ethereum/portal-network-specs/blob/master/testnet.md#bootnodes).
It should be noted that there is currently no real way to distinguish a "specific" Portal
network, and as long as the same Portal protocols are supported, nodes can
simply connect to it and no real separation can be made.
When testing locally the `--network:none` option can be provided to avoid
connecting to any of the testnet bootstrap nodes.
<!-- TODO: Update this once we have the headersWithProof type merged and data on the network -->
The Portal testnet is slowly being filled up with historical data through bridge nodes.
Because of this, more recent history data is more likely to be available.. This can
be tested by using the JSON-RPC call `eth_getBlockByHash`:
```
# Get the hash of a block from your favorite block explorer, e.g.:
BLOCKHASH=0x34eea44911b19f9aa8c72f69bdcbda3ed933b11a940511b6f3f58a87427231fb # Replace this to the block hash of your choice
# Run this command to get this block:
curl -s -X POST -H 'Content-Type: application/json' -d '{"jsonrpc":"2.0","id":"1","method":"eth_getBlockByHash","params":["'${BLOCKHASH}'", true]}' http://localhost:8545 | jq
```
One can also use the `blockwalk` tool to walk down the blocks one by one, e.g:
```bash
make blockwalk
BLOCKHASH=0x34eea44911b19f9aa8c72f69bdcbda3ed933b11a940511b6f3f58a87427231fb # Replace this to the block hash of your choice
./build/blockwalk --block-hash:${BLOCKHASH}
```
### Run Fluffy test suite
```bash
# From the nimbus-eth1 repository
make fluffy-test
```
### Run Fluffy local testnet script
```bash
./fluffy/scripts/launch_local_testnet.sh
```
Find more details on the usage and workings of the local testnet script
[here](./docs/local_testnet.md).
### Windows support
Follow the steps outlined [here](../README.md#windows) to build fluffy on Windows.
## The basics for developers
When working on this repository, you can run the `env.sh` script to run a
command with the right environment variables set. This means the vendored
Nim and Nim modules will be used, just as when you use `make`.
E.g.:
```bash
# start a new interactive shell with the right env vars set
./env.sh bash
```
More [development tips](../README.md#devel-tips)
can be found on the general nimbus-eth1 readme.
The code follows the
[Status Nim Style Guide](https://status-im.github.io/nim-style-guide/).
## Build local dev container for portal-hive
To develop code against portal-hive tests you will need:
1) Clone and build portal-hive ([#1](https://github.com/ethereum/portal-hive))
2) Modify `Dockerfile` for fluffy in `portal-hive/clients/fluffy/Dockerfile` ([#2](https://github.com/ethereum/portal-hive/blob/main/docs/overview.md#running-a-client-built-from-source))
3) Build local dev container using following command: ```docker build --tag fluffy-dev --file ./fluffy/tools/docker/Dockerfile.portalhive .``` You may need to change fluffy-dev to the tag you using in portal-hive client dockerfile.
4) Run the tests
Also keep in mind that `./vendors` is dockerignored and cached. If you have to make local changes to one of the dependencies in that directory you'll have to remove `vendors/` from `./fluffy/tools/docker/Dockerfile.portalhive.dockerignore`.
## Metrics and their visualisation
To enable metrics run Fluffy with the `--metrics` flag:
```bash
./build/fluffy --metrics
```
Default the metrics are available at http://127.0.0.1:8008/metrics.
The address can be changed with the `--metrics-address` and `--metrics-port` options.
This provides only a snapshot of the current metrics. In order track the metrics over
time and to also visualise them one can use for example Prometheus and Grafana.
The steps on how to set up such system is explained in [this guide](https://nimbus.guide/metrics-pretty-pictures.html#prometheus-and-grafana).
A Fluffy specific dashboard can be found [here](./grafana/fluffy_grafana_dashboard.json).
This is the dashboard used for our Fluffy testnet fleet.
In order to use it locally, you will have to remove the
`{job="nimbus-fluffy-metrics"}` part from the `instance` and `container`
variables queries in the dashboard settings. Or they can also be changed to a
constant value.
The other option would be to remove those variables and remove their usage in
each panel query.
## License
Licensed and distributed under either of

View File

@ -0,0 +1,32 @@
# Access content on the Portal network
Once you have a Fluffy node [connected to network](./connect-to-portal.md) with
the JSON-RPC interface enabled, then you can access the content available on
the Portal network.
You can for example access execution layer blocks through the standardized
JSON-RPC call `eth_getBlockByHash`:
```bash
# Get the hash of a block from your favorite block explorer, e.g.:
BLOCKHASH=0x34eea44911b19f9aa8c72f69bdcbda3ed933b11a940511b6f3f58a87427231fb # Replace this to the block hash of your choice
# Run this command to get this block:
curl -s -X POST -H 'Content-Type: application/json' -d '{"jsonrpc":"2.0","id":"1","method":"eth_getBlockByHash","params":["'${BLOCKHASH}'", true]}' http://localhost:8545 | jq
```
!!! note
The Portal testnet is slowly being filled up with historical data through
bridge nodes. Because of this, more recent history data is more likely to be
available.
You can also use our `blockwalk` tool to walk down the blocks one by one:
```bash
make blockwalk
BLOCKHASH=0x34eea44911b19f9aa8c72f69bdcbda3ed933b11a940511b6f3f58a87427231fb # Replace this to the block hash of your choice
./build/blockwalk --block-hash:${BLOCKHASH}
```
<!-- TODO: Give some examples of direct portal content access with the Portal API
and about recently audited data at http://glados.ethportal.net/content/ -->

View File

@ -0,0 +1,21 @@
# The basics for developers
When working on Fluffy in the nimbus-eth1 repository, you can run the `env.sh`
script to run a command with the right environment variables set. This means the
vendored Nim and Nim modules will be used, just as when you use `make`.
E.g.:
```bash
# start a new interactive shell with the right env vars set
./env.sh bash
```
<!-- TODO: Add most important development tips from following page here and
remove the link -->
More [development tips](https://github.com/status-im/nimbus-eth1/README.md#devel-tips)
can be found on the general nimbus-eth1 readme.
The code follows the
[Status Nim Style Guide](https://status-im.github.io/nim-style-guide/).

View File

@ -0,0 +1,48 @@
# Build from source
Building Fluffy from source ensures that all hardware-specific optimizations are
turned on.
The build process itself is simple and fully automated, but may take a few minutes.
!!! note "Nim"
Fluffy is written in the [Nim](https://nim-lang.org) programming language.
The correct version will automatically be downloaded as part of the build process!
## Prerequisites
Make sure you have all needed [prerequisites](./prerequisites.md).
## Building the Fluffy client
### 1. Clone the `nimbus-eth1` repository
```sh
git clone git@github.com:status-im/nimbus-eth1.git
cd nimbus-eth1
```
### 2. Run the Fluffy build process
To build Fluffy and its dependencies, run:
```sh
make fluffy
```
This step can take several minutes.
After it has finished, you can check if the build was successful by running:
```sh
# See available command line options
./build/fluffy --help
```
If you see the command-line options, your installation was successful!
Otherwise, don't hesitate to reach out to us in the `#nimbus-fluffy` channel of
[our discord](https://discord.gg/j3nYBUeEad).
## Keeping Fluffy updated
When you decide to upgrade Fluffy to a newer version, make sure to follow the
[how to upgrade page](./upgrade.md).

View File

@ -0,0 +1,35 @@
# Connect to the Portal network
Connecting to the current Portal network is as easy as running following command:
```sh
./build/fluffy --rpc
```
This will connect to the public [Portal testnet](https://github.com/ethereum/portal-network-specs/blob/master/testnet.md#portal-network-testnet)
which contains nodes of the different clients.
!!! note
Default the Fluffy node will connect to the
[bootstrap nodes](https://github.com/ethereum/portal-network-specs/blob/master/testnet.md#bootnodes) of the public testnet.
When testing locally the `--network:none` option can be provided to avoid
connecting to any of the testnet bootstrap nodes.
The `--rpc` option will also enable the different JSON-RPC interfaces through
which you can access the Portal Network.
Fluffy fully supports the [Portal Network JSON-RPC Specification](https://playground.open-rpc.org/?schemaUrl=https://raw.githubusercontent.com/ethereum/portal-network-specs/assembled-spec/jsonrpc/openrpc.json&uiSchema%5BappBar%5D%5Bui:splitView%5D=false&uiSchema%5BappBar%5D%5Bui:input%5D=false&uiSchema%5BappBar%5D%5Bui:examplesDropdown%5D=false).
Fluffy also supports a small subset of the [Execution JSON-RPC API](https://ethereum.github.io/execution-apis/api-documentation/).
!!! note
The end goal is to be able to fully support the Execution JSON-RPC API,
however currently not all Portal networks are specified, implemented or
rolled out to be able to provide this.
<!-- TODO: Explain some of the more important cli options here? Or in a separate
page? -->

View File

@ -0,0 +1,58 @@
# Fluffy with Portal-hive
Fluffy is one of the Portal clients that is being tested with [portal-hive](https://github.com/ethereum/portal-hive).
To see the status of the tests for the current version you can access [https://portal-hive.ethdevops.io/](https://portal-hive.ethdevops.io/).
## Run the hive tests locally
Build portal-hive:
```sh
git clone https://github.com/ethereum/portal-hive.git
cd ./portal-hive
go build .
```
Example commands for running test suites:
```sh
# Run the rpc-compat tests with the 3 different clients
./hive --sim rpc-compat --client fluffy,trin,ultralight
# Run the portal-interop tests with only the fluffy client
./hive --sim portal-interop --client fluffy
# Access results through web-ui:
```sh
go build ./cmd/hiveview
./hiveview --serve --logdir ./workspace/logs
```
!!! note
You can see all the implemented simulators in [https://github.com/ethereum/portal-hive/tree/main/simulators](https://github.com/ethereum/portal-hive/tree/main/simulators)
## Build a local development Docker image for portal-hive
To debug & develop Fluffy code against portal-hive tests you might want to
create a local development docker image for Fluffy.
To do that follow next steps:
1) Clone and build portal-hive, see above.
2) Build the local development Docker image using the following command:
```
docker build --tag fluffy-dev --file ./fluffy/tools/docker/Dockerfile.portalhive .
```
3) Modify the `FROM` tag in the portal-hive `Dockerfile` of fluffy at
`portal-hive/clients/fluffy/Dockerfile` to use the image that was buid in step 2.
4) Run the tests as [usually](fluffy-with-portal-hive.md/#run-the-hive-tests-locally).
!!! warning
The `./vendors` dir is dockerignored and cached. If you have to make local
changes to one of the dependencies in that directory you will have to remove
`vendors/` from `./fluffy/tools/docker/Dockerfile.portalhive.dockerignore`.

View File

@ -0,0 +1,36 @@
# Metrics and their visualisation
In this page we'll cover how to enable metrics and how to use Grafana and
Prometheus to help you visualize these real-time metrics concerning the Fluffy
node.
## Enable metrics in Fluffy
To enable metrics run Fluffy with the `--metrics` flag:
```bash
./build/fluffy --metrics
```
Default the metrics are available at [http://127.0.0.1:8008/metrics](http://127.0.0.1:8008/metrics).
The address can be changed with the `--metrics-address` and `--metrics-port` options.
This provides only a snapshot of the current metrics. In order track the metrics
over time and to also visualise them one can use for example Prometheus and Grafana.
## Visualisation through Prometheus and Grafana
<!-- TODO: Rework this page without linking to nimbus.guide page about metrics -->
The steps on how to set up metrics visualisation with Prometheus and Grafana is
explained in [this guide](https://nimbus.guide/metrics-pretty-pictures.html#prometheus-and-grafana).
A Fluffy specific dashboard can be found [here](https://github.com/status-im/nimbus-eth1/blob/master/fluffy/grafana/fluffy_grafana_dashboard.json).
This is the dashboard used for our Fluffy testnet fleet.
In order to use it locally, you will have to remove the
`{job="nimbus-fluffy-metrics"}` part from the `instance` and `container`
variables queries in the dashboard settings. Or they can also be changed to a
constant value.
The other option would be to remove those variables and remove their usage in
each panel query.

View File

@ -0,0 +1,65 @@
# Prerequisites
The Fluffy client runs on Linux, macOS, Windows, and Android.
## Build prerequisites
When building from source, you will need additional build dependencies to be
installed:
- Developer tools (C compiler, Make, Bash, Git 2.9.4 or newer)
- [CMake](https://cmake.org/)
=== "Linux"
On common Linux distributions the dependencies can be installed with:
```sh
# Debian and Ubuntu
sudo apt-get install build-essential git cmake
# Fedora
dnf install @development-tools cmake
# Arch Linux, using an AUR manager
yourAURmanager -S base-devel cmake
```
=== "macOS"
With [Homebrew](https://brew.sh/):
```sh
brew install cmake
```
=== "Windows"
To build Fluffy on Windows, the MinGW-w64 build environment is recommended.
- Install Mingw-w64 for your architecture using the "[MinGW-W64 Online Installer](https://sourceforge.net/projects/mingw-w64/files/)":
1. Select your architecture in the setup menu (`i686` on 32-bit, `x86_64` on 64-bit).
2. Set threads to `win32`.
3. Set exceptions to "dwarf" on 32-bit and "seh" on 64-bit.
4. Change the installation directory to `C:\mingw-w64` and add it to your system PATH in `"My Computer"/"This PC" -> Properties -> Advanced system settings -> Environment Variables -> Path -> Edit -> New -> C:\mingw-w64\mingw64\bin` (`C:\mingw-w64\mingw32\bin` on 32-bit).
!!! note
If the online installer isn't working you can try installing `mingw-w64` through [MSYS2](https://www.msys2.org/).
- Install [cmake](https://cmake.org/).
- Install [Git for Windows](https://gitforwindows.org/) and use a "Git Bash"
shell to clone nimbus-eth1 and build fluffy.
=== "Android"
- Install the [Termux](https://termux.com) app from FDroid or the Google Play store
- Install a [PRoot](https://wiki.termux.com/wiki/PRoot) of your choice following the instructions for your preferred distribution.
The Ubuntu PRoot is known to contain all Fluffy prerequisites compiled on Arm64 architecture (the most common architecture for Android devices).
Assuming you use Ubuntu PRoot:
```sh
apt install build-essential git
```

View File

@ -0,0 +1,15 @@
# Fluffy test suite
## Run Fluffy test suite
```bash
# From the nimbus-eth1 repository
make fluffy-test
```
## Run Fluffy local testnet script
```bash
./fluffy/scripts/launch_local_testnet.sh --run-tests
```
Find more details on the usage and workings of the local testnet script
[here](./run-local-testnet.md).

View File

@ -0,0 +1,28 @@
# Upgrade
To upgrade to the latest version you need to update the nimbus-eth1 repository
and re-compile Fluffy.
!!! note
In this state of development there are no official releases yet nor git tags
for different versions.
## Upgrade to the latest version
Upgrading Fluffy when built from source is similar to the installation process.
Run:
```sh
# Download the updated source code
git pull && make update
# Build Fluffy from the newly updated source
make -j4 fluffy
```
Complete the upgrade by restarting the node.
!!! tip
To check which version of Fluffy you're currently running, run
`./build/fluffy --version`

View File

@ -21,7 +21,8 @@ theme:
# Support urls previously used by mdbook
use_directory_urls: false
repo_url: https://github.com/status-im/nimbus-eth1
edit_uri: edit/unstable/docs/the_nimbus_book/src
edit_uri: edit/master/fluffy/docs/the_fluffy_book/docs
docs_dir: docs
markdown_extensions:
- admonition
@ -49,14 +50,25 @@ nav:
- 'quick-start-docker.md'
- How-to:
# - 'install.md'
# - 'build.md'
- Installation:
- 'prerequisites.md'
- 'build-from-source.md'
- 'upgrade.md'
- 'metrics.md'
- Access the network:
- 'connect-to-portal.md'
- 'access-content.md'
- Portal bridge:
- 'history-content-bridging.md'
- 'beacon-content-bridging.md'
- Developers:
- 'basics-for-developers.md'
- 'test-suite.md'
- Local testnet:
- 'run-local-testnet.md'
- 'testnet-history-network.md'
- 'testnet-beacon-network.md'
- 'fluffy-with-portal-hive.md'
- 'protocol-interop-testing.md'
- 'eth-data-exporter.md'