diff --git a/fluffy/README.md b/fluffy/README.md index e8cb04e19..a68936629 100644 --- a/fluffy/README.md +++ b/fluffy/README.md @@ -9,13 +9,15 @@ [![Status: #nimbus-general](https://img.shields.io/badge/Status-nimbus--general-blue.svg)](https://join.status.im/nimbus-general) ## Introduction -This folder holds the development of the Nimbus client implementation supporting -the Portal Network: Fluffy. The Portal Network is a project still heavily in -research phase and fully in flux. This client is thus still highly experimental. -Current status of specifications can be found in the -[portal-network-specs repository](https://github.com/ethereum/portal-network-specs/blob/master/portal-network.md). +Fluffy is the Nimbus client implementation of the +[Portal network specifications](https://github.com/ethereum/portal-network-specs/blob/master/portal-network.md). +You can find all the information you need to run a Fluffy node in +[The Fluffy Book](https://status-im.github.io/nimbus-eth1). + +The [quickstart page](https://status-im.github.io/nimbus-eth1/quick-start.html) +in particular will help you to get quickly connected to the Portal network. ## Development Updates @@ -25,163 +27,6 @@ Monthly development updates are shared To keep up to date with changes and development progress, follow the [Nimbus blog](https://our.status.im/tag/nimbus/). -## How to Build & Run - -### Prerequisites -- GNU Make, Bash and the usual POSIX utilities. Git 2.9.4 or newer. - -### Build the Fluffy client -```bash -git clone git@github.com:status-im/nimbus-eth1.git -cd nimbus-eth1 -make fluffy - -# See available command line options -./build/fluffy --help - -# Example command: Run the client and connect to a bootstrap node. -./build/fluffy --bootstrap-node:enr: -``` - -### Update and rebuild the Fluffy client -```bash -# From the nimbus-eth1 repository -git pull -# To bring the git submodules up to date -make update - -make fluffy -``` - -### Run a Fluffy client on the public testnet - -```bash -# Connect to the Portal testnet bootstrap nodes and enable the JSON-RPC APIs -./build/fluffy --rpc --table-ip-limit:1024 --bucket-ip-limit:24 -``` - -The `table-ip-limit` and `bucket-ip-limit` options are needed to allow more -nodes with the same IPs in the routing tables. The default limits are there -as security measure. It is currently needed to increase the limits for the testnet -because the fleet of Fluffy nodes runs on only 2 machines / network interfaces. - -There is a public [Portal testnet](https://github.com/ethereum/portal-network-specs/blob/master/testnet.md#portal-network-testnet) -which contains nodes of different clients. - -Fluffy will default join the network through these bootstrap nodes. -You can also explicitly provide the `--network:testnet0` option to join this -network, or `--network:none` to not connect to any of these bootstrapn odes. - -> **_Note:_** The `--network` option selects automatically a static set of -specific bootstrap nodes belonging to a "testnet". Currently `testnet0` is the -only option, which results in connecting to the -[testnet bootstrap nodes](https://github.com/ethereum/portal-network-specs/blob/master/testnet.md#bootnodes). -It should be noted that there is currently no real way to distinguish a "specific" Portal -network, and as long as the same Portal protocols are supported, nodes can -simply connect to it and no real separation can be made. -When testing locally the `--network:none` option can be provided to avoid -connecting to any of the testnet bootstrap nodes. - - - -The Portal testnet is slowly being filled up with historical data through bridge nodes. -Because of this, more recent history data is more likely to be available.. This can -be tested by using the JSON-RPC call `eth_getBlockByHash`: -``` -# Get the hash of a block from your favorite block explorer, e.g.: -BLOCKHASH=0x34eea44911b19f9aa8c72f69bdcbda3ed933b11a940511b6f3f58a87427231fb # Replace this to the block hash of your choice -# Run this command to get this block: -curl -s -X POST -H 'Content-Type: application/json' -d '{"jsonrpc":"2.0","id":"1","method":"eth_getBlockByHash","params":["'${BLOCKHASH}'", true]}' http://localhost:8545 | jq -``` - -One can also use the `blockwalk` tool to walk down the blocks one by one, e.g: -```bash -make blockwalk - -BLOCKHASH=0x34eea44911b19f9aa8c72f69bdcbda3ed933b11a940511b6f3f58a87427231fb # Replace this to the block hash of your choice -./build/blockwalk --block-hash:${BLOCKHASH} -``` - -### Run Fluffy test suite -```bash -# From the nimbus-eth1 repository -make fluffy-test -``` - -### Run Fluffy local testnet script -```bash -./fluffy/scripts/launch_local_testnet.sh -``` - -Find more details on the usage and workings of the local testnet script -[here](./docs/local_testnet.md). - -### Windows support - -Follow the steps outlined [here](../README.md#windows) to build fluffy on Windows. - - -## The basics for developers - -When working on this repository, you can run the `env.sh` script to run a -command with the right environment variables set. This means the vendored -Nim and Nim modules will be used, just as when you use `make`. - -E.g.: - -```bash -# start a new interactive shell with the right env vars set -./env.sh bash -``` - -More [development tips](../README.md#devel-tips) -can be found on the general nimbus-eth1 readme. - -The code follows the -[Status Nim Style Guide](https://status-im.github.io/nim-style-guide/). - - -## Build local dev container for portal-hive - -To develop code against portal-hive tests you will need: - -1) Clone and build portal-hive ([#1](https://github.com/ethereum/portal-hive)) - -2) Modify `Dockerfile` for fluffy in `portal-hive/clients/fluffy/Dockerfile` ([#2](https://github.com/ethereum/portal-hive/blob/main/docs/overview.md#running-a-client-built-from-source)) - -3) Build local dev container using following command: ```docker build --tag fluffy-dev --file ./fluffy/tools/docker/Dockerfile.portalhive .``` You may need to change fluffy-dev to the tag you using in portal-hive client dockerfile. - -4) Run the tests - -Also keep in mind that `./vendors` is dockerignored and cached. If you have to make local changes to one of the dependencies in that directory you'll have to remove `vendors/` from `./fluffy/tools/docker/Dockerfile.portalhive.dockerignore`. - - -## Metrics and their visualisation - -To enable metrics run Fluffy with the `--metrics` flag: -```bash -./build/fluffy --metrics -``` -Default the metrics are available at http://127.0.0.1:8008/metrics. - -The address can be changed with the `--metrics-address` and `--metrics-port` options. - -This provides only a snapshot of the current metrics. In order track the metrics over -time and to also visualise them one can use for example Prometheus and Grafana. - -The steps on how to set up such system is explained in [this guide](https://nimbus.guide/metrics-pretty-pictures.html#prometheus-and-grafana). - -A Fluffy specific dashboard can be found [here](./grafana/fluffy_grafana_dashboard.json). - -This is the dashboard used for our Fluffy testnet fleet. -In order to use it locally, you will have to remove the -`{job="nimbus-fluffy-metrics"}` part from the `instance` and `container` -variables queries in the dashboard settings. Or they can also be changed to a -constant value. - -The other option would be to remove those variables and remove their usage in -each panel query. - ## License Licensed and distributed under either of diff --git a/fluffy/docs/the_fluffy_book/docs/access-content.md b/fluffy/docs/the_fluffy_book/docs/access-content.md new file mode 100644 index 000000000..b756f4d11 --- /dev/null +++ b/fluffy/docs/the_fluffy_book/docs/access-content.md @@ -0,0 +1,32 @@ +# Access content on the Portal network + +Once you have a Fluffy node [connected to network](./connect-to-portal.md) with +the JSON-RPC interface enabled, then you can access the content available on +the Portal network. + +You can for example access execution layer blocks through the standardized +JSON-RPC call `eth_getBlockByHash`: + +```bash +# Get the hash of a block from your favorite block explorer, e.g.: +BLOCKHASH=0x34eea44911b19f9aa8c72f69bdcbda3ed933b11a940511b6f3f58a87427231fb # Replace this to the block hash of your choice +# Run this command to get this block: +curl -s -X POST -H 'Content-Type: application/json' -d '{"jsonrpc":"2.0","id":"1","method":"eth_getBlockByHash","params":["'${BLOCKHASH}'", true]}' http://localhost:8545 | jq +``` + +!!! note + The Portal testnet is slowly being filled up with historical data through + bridge nodes. Because of this, more recent history data is more likely to be + available. + +You can also use our `blockwalk` tool to walk down the blocks one by one: +```bash +make blockwalk + +BLOCKHASH=0x34eea44911b19f9aa8c72f69bdcbda3ed933b11a940511b6f3f58a87427231fb # Replace this to the block hash of your choice +./build/blockwalk --block-hash:${BLOCKHASH} +``` + + + \ No newline at end of file diff --git a/fluffy/docs/the_fluffy_book/docs/basics-for-developers.md b/fluffy/docs/the_fluffy_book/docs/basics-for-developers.md new file mode 100644 index 000000000..feeaa2250 --- /dev/null +++ b/fluffy/docs/the_fluffy_book/docs/basics-for-developers.md @@ -0,0 +1,21 @@ +# The basics for developers + +When working on Fluffy in the nimbus-eth1 repository, you can run the `env.sh` +script to run a command with the right environment variables set. This means the +vendored Nim and Nim modules will be used, just as when you use `make`. + +E.g.: + +```bash +# start a new interactive shell with the right env vars set +./env.sh bash +``` + + + +More [development tips](https://github.com/status-im/nimbus-eth1/README.md#devel-tips) +can be found on the general nimbus-eth1 readme. + +The code follows the +[Status Nim Style Guide](https://status-im.github.io/nim-style-guide/). diff --git a/fluffy/docs/the_fluffy_book/docs/build-from-source.md b/fluffy/docs/the_fluffy_book/docs/build-from-source.md new file mode 100644 index 000000000..d3dc04c9c --- /dev/null +++ b/fluffy/docs/the_fluffy_book/docs/build-from-source.md @@ -0,0 +1,48 @@ +# Build from source + +Building Fluffy from source ensures that all hardware-specific optimizations are +turned on. +The build process itself is simple and fully automated, but may take a few minutes. + +!!! note "Nim" + Fluffy is written in the [Nim](https://nim-lang.org) programming language. + The correct version will automatically be downloaded as part of the build process! + +## Prerequisites + +Make sure you have all needed [prerequisites](./prerequisites.md). + +## Building the Fluffy client + +### 1. Clone the `nimbus-eth1` repository + +```sh +git clone git@github.com:status-im/nimbus-eth1.git +cd nimbus-eth1 +``` + +### 2. Run the Fluffy build process + +To build Fluffy and its dependencies, run: + +```sh +make fluffy +``` + +This step can take several minutes. +After it has finished, you can check if the build was successful by running: + +```sh +# See available command line options +./build/fluffy --help +``` + +If you see the command-line options, your installation was successful! +Otherwise, don't hesitate to reach out to us in the `#nimbus-fluffy` channel of +[our discord](https://discord.gg/j3nYBUeEad). + + +## Keeping Fluffy updated + +When you decide to upgrade Fluffy to a newer version, make sure to follow the +[how to upgrade page](./upgrade.md). \ No newline at end of file diff --git a/fluffy/docs/the_fluffy_book/docs/connect-to-portal.md b/fluffy/docs/the_fluffy_book/docs/connect-to-portal.md new file mode 100644 index 000000000..9eeeb434f --- /dev/null +++ b/fluffy/docs/the_fluffy_book/docs/connect-to-portal.md @@ -0,0 +1,35 @@ +# Connect to the Portal network + +Connecting to the current Portal network is as easy as running following command: + +```sh +./build/fluffy --rpc +``` + +This will connect to the public [Portal testnet](https://github.com/ethereum/portal-network-specs/blob/master/testnet.md#portal-network-testnet) +which contains nodes of the different clients. + +!!! note + Default the Fluffy node will connect to the + [bootstrap nodes](https://github.com/ethereum/portal-network-specs/blob/master/testnet.md#bootnodes) of the public testnet. + + When testing locally the `--network:none` option can be provided to avoid + connecting to any of the testnet bootstrap nodes. + +The `--rpc` option will also enable the different JSON-RPC interfaces through +which you can access the Portal Network. + +Fluffy fully supports the [Portal Network JSON-RPC Specification](https://playground.open-rpc.org/?schemaUrl=https://raw.githubusercontent.com/ethereum/portal-network-specs/assembled-spec/jsonrpc/openrpc.json&uiSchema%5BappBar%5D%5Bui:splitView%5D=false&uiSchema%5BappBar%5D%5Bui:input%5D=false&uiSchema%5BappBar%5D%5Bui:examplesDropdown%5D=false). + +Fluffy also supports a small subset of the [Execution JSON-RPC API](https://ethereum.github.io/execution-apis/api-documentation/). + + +!!! note + The end goal is to be able to fully support the Execution JSON-RPC API, + however currently not all Portal networks are specified, implemented or + rolled out to be able to provide this. + + + + diff --git a/fluffy/docs/the_fluffy_book/docs/fluffy-with-portal-hive.md b/fluffy/docs/the_fluffy_book/docs/fluffy-with-portal-hive.md new file mode 100644 index 000000000..5b39949a2 --- /dev/null +++ b/fluffy/docs/the_fluffy_book/docs/fluffy-with-portal-hive.md @@ -0,0 +1,58 @@ +# Fluffy with Portal-hive + +Fluffy is one of the Portal clients that is being tested with [portal-hive](https://github.com/ethereum/portal-hive). + +To see the status of the tests for the current version you can access [https://portal-hive.ethdevops.io/](https://portal-hive.ethdevops.io/). + +## Run the hive tests locally + +Build portal-hive: + +```sh +git clone https://github.com/ethereum/portal-hive.git +cd ./portal-hive +go build . +``` + +Example commands for running test suites: + +```sh +# Run the rpc-compat tests with the 3 different clients +./hive --sim rpc-compat --client fluffy,trin,ultralight + +# Run the portal-interop tests with only the fluffy client +./hive --sim portal-interop --client fluffy + +# Access results through web-ui: +```sh +go build ./cmd/hiveview +./hiveview --serve --logdir ./workspace/logs +``` + +!!! note + You can see all the implemented simulators in [https://github.com/ethereum/portal-hive/tree/main/simulators](https://github.com/ethereum/portal-hive/tree/main/simulators) + +## Build a local development Docker image for portal-hive + +To debug & develop Fluffy code against portal-hive tests you might want to +create a local development docker image for Fluffy. + +To do that follow next steps: + +1) Clone and build portal-hive, see above. + +2) Build the local development Docker image using the following command: +``` +docker build --tag fluffy-dev --file ./fluffy/tools/docker/Dockerfile.portalhive . +``` + +3) Modify the `FROM` tag in the portal-hive `Dockerfile` of fluffy at +`portal-hive/clients/fluffy/Dockerfile` to use the image that was buid in step 2. + +4) Run the tests as [usually](fluffy-with-portal-hive.md/#run-the-hive-tests-locally). + +!!! warning + The `./vendors` dir is dockerignored and cached. If you have to make local + changes to one of the dependencies in that directory you will have to remove + `vendors/` from `./fluffy/tools/docker/Dockerfile.portalhive.dockerignore`. + diff --git a/fluffy/docs/the_fluffy_book/docs/metrics.md b/fluffy/docs/the_fluffy_book/docs/metrics.md new file mode 100644 index 000000000..bf841d0cb --- /dev/null +++ b/fluffy/docs/the_fluffy_book/docs/metrics.md @@ -0,0 +1,36 @@ +# Metrics and their visualisation + +In this page we'll cover how to enable metrics and how to use Grafana and +Prometheus to help you visualize these real-time metrics concerning the Fluffy +node. + +## Enable metrics in Fluffy + +To enable metrics run Fluffy with the `--metrics` flag: +```bash +./build/fluffy --metrics +``` +Default the metrics are available at [http://127.0.0.1:8008/metrics](http://127.0.0.1:8008/metrics). + +The address can be changed with the `--metrics-address` and `--metrics-port` options. + +This provides only a snapshot of the current metrics. In order track the metrics +over time and to also visualise them one can use for example Prometheus and Grafana. + +## Visualisation through Prometheus and Grafana + + + +The steps on how to set up metrics visualisation with Prometheus and Grafana is +explained in [this guide](https://nimbus.guide/metrics-pretty-pictures.html#prometheus-and-grafana). + +A Fluffy specific dashboard can be found [here](https://github.com/status-im/nimbus-eth1/blob/master/fluffy/grafana/fluffy_grafana_dashboard.json). + +This is the dashboard used for our Fluffy testnet fleet. +In order to use it locally, you will have to remove the +`{job="nimbus-fluffy-metrics"}` part from the `instance` and `container` +variables queries in the dashboard settings. Or they can also be changed to a +constant value. + +The other option would be to remove those variables and remove their usage in +each panel query. diff --git a/fluffy/docs/the_fluffy_book/docs/prerequisites.md b/fluffy/docs/the_fluffy_book/docs/prerequisites.md new file mode 100644 index 000000000..0324c5049 --- /dev/null +++ b/fluffy/docs/the_fluffy_book/docs/prerequisites.md @@ -0,0 +1,65 @@ +# Prerequisites + +The Fluffy client runs on Linux, macOS, Windows, and Android. + +## Build prerequisites + +When building from source, you will need additional build dependencies to be +installed: + +- Developer tools (C compiler, Make, Bash, Git 2.9.4 or newer) +- [CMake](https://cmake.org/) + +=== "Linux" + + On common Linux distributions the dependencies can be installed with: + + ```sh + # Debian and Ubuntu + sudo apt-get install build-essential git cmake + + # Fedora + dnf install @development-tools cmake + + # Arch Linux, using an AUR manager + yourAURmanager -S base-devel cmake + ``` + +=== "macOS" + + With [Homebrew](https://brew.sh/): + + ```sh + brew install cmake + ``` + +=== "Windows" + To build Fluffy on Windows, the MinGW-w64 build environment is recommended. + + - Install Mingw-w64 for your architecture using the "[MinGW-W64 Online Installer](https://sourceforge.net/projects/mingw-w64/files/)": + + 1. Select your architecture in the setup menu (`i686` on 32-bit, `x86_64` on 64-bit). + 2. Set threads to `win32`. + 3. Set exceptions to "dwarf" on 32-bit and "seh" on 64-bit. + 4. Change the installation directory to `C:\mingw-w64` and add it to your system PATH in `"My Computer"/"This PC" -> Properties -> Advanced system settings -> Environment Variables -> Path -> Edit -> New -> C:\mingw-w64\mingw64\bin` (`C:\mingw-w64\mingw32\bin` on 32-bit). + + !!! note + If the online installer isn't working you can try installing `mingw-w64` through [MSYS2](https://www.msys2.org/). + + - Install [cmake](https://cmake.org/). + + - Install [Git for Windows](https://gitforwindows.org/) and use a "Git Bash" + shell to clone nimbus-eth1 and build fluffy. + + +=== "Android" + + - Install the [Termux](https://termux.com) app from FDroid or the Google Play store + - Install a [PRoot](https://wiki.termux.com/wiki/PRoot) of your choice following the instructions for your preferred distribution. + The Ubuntu PRoot is known to contain all Fluffy prerequisites compiled on Arm64 architecture (the most common architecture for Android devices). + + Assuming you use Ubuntu PRoot: + + ```sh + apt install build-essential git + ``` diff --git a/fluffy/docs/the_fluffy_book/docs/test-suite.md b/fluffy/docs/the_fluffy_book/docs/test-suite.md new file mode 100644 index 000000000..a4b90e8b7 --- /dev/null +++ b/fluffy/docs/the_fluffy_book/docs/test-suite.md @@ -0,0 +1,15 @@ +# Fluffy test suite + +## Run Fluffy test suite +```bash +# From the nimbus-eth1 repository +make fluffy-test +``` + +## Run Fluffy local testnet script +```bash +./fluffy/scripts/launch_local_testnet.sh --run-tests +``` + +Find more details on the usage and workings of the local testnet script +[here](./run-local-testnet.md). diff --git a/fluffy/docs/the_fluffy_book/docs/upgrade.md b/fluffy/docs/the_fluffy_book/docs/upgrade.md new file mode 100644 index 000000000..d51dc1f32 --- /dev/null +++ b/fluffy/docs/the_fluffy_book/docs/upgrade.md @@ -0,0 +1,28 @@ +# Upgrade + +To upgrade to the latest version you need to update the nimbus-eth1 repository +and re-compile Fluffy. + +!!! note + In this state of development there are no official releases yet nor git tags + for different versions. + +## Upgrade to the latest version + +Upgrading Fluffy when built from source is similar to the installation process. + +Run: + +```sh +# Download the updated source code +git pull && make update + +# Build Fluffy from the newly updated source +make -j4 fluffy +``` + +Complete the upgrade by restarting the node. + +!!! tip + To check which version of Fluffy you're currently running, run + `./build/fluffy --version` diff --git a/fluffy/docs/the_fluffy_book/mkdocs.yml b/fluffy/docs/the_fluffy_book/mkdocs.yml index a1f57924d..ce02acbc7 100644 --- a/fluffy/docs/the_fluffy_book/mkdocs.yml +++ b/fluffy/docs/the_fluffy_book/mkdocs.yml @@ -21,7 +21,8 @@ theme: # Support urls previously used by mdbook use_directory_urls: false repo_url: https://github.com/status-im/nimbus-eth1 -edit_uri: edit/unstable/docs/the_nimbus_book/src +edit_uri: edit/master/fluffy/docs/the_fluffy_book/docs +docs_dir: docs markdown_extensions: - admonition @@ -49,14 +50,25 @@ nav: - 'quick-start-docker.md' - How-to: - # - 'install.md' - # - 'build.md' + - Installation: + - 'prerequisites.md' + - 'build-from-source.md' + - 'upgrade.md' + - 'metrics.md' + - Access the network: + - 'connect-to-portal.md' + - 'access-content.md' - Portal bridge: - 'history-content-bridging.md' - 'beacon-content-bridging.md' + + - Developers: + - 'basics-for-developers.md' + - 'test-suite.md' - Local testnet: - 'run-local-testnet.md' - 'testnet-history-network.md' - 'testnet-beacon-network.md' + - 'fluffy-with-portal-hive.md' - 'protocol-interop-testing.md' - 'eth-data-exporter.md'