Book updates end may (#2629)

* break up long commands, geth > infura

* fill out profitability page

* fill in 'increase privacy'

* update systemd

* add tersec's restarting loop

* update privacy page

* consolidate eth1 instructions

* tidy up eth1.md

* fill out health.md

* rm contribute-network.md from toc

* edit eth1.md

* incorporate review feedback modulo profits.md

* update health.md

* remove script to monitor logs

docs/the_nimbus_book/src/SUMMARY.md

@@ -34,6 +34,8 @@
 - [Back up your database](./database-backup.md)
 - [Set up email notifications](./email-notifications.md)
 - [Graffiti the blockchain](./graffiti.md)
+- [Optimise for profitability](./profits.md)
+- [Monitor the health of your node](./health.md)
 
 # Guides
 - [Grafana and Prometheus](./metrics-pretty-pictures.md)

docs/the_nimbus_book/src/beacon-node-systemd.md

@@ -67,6 +67,10 @@ sudo journalctl -u nimbus-eth2-prater.service
 
 This will show you the Nimbus logs at the default setting -- it should include regular "slot start" messages which will show your [sync progress](./keep-an-eye.md#keep-track-of-your-syncing-progress).
 
+To rewind logs - by one day, say - run:
+
+```console
+sudo journalctl -u nimbus-eth2-prater.service --since yesterday
+```
+
 For more options, see [here](https://www.raspberrypi.org/documentation/linux/usage/systemd.md).
-
-

docs/the_nimbus_book/src/contribute.md

@@ -8,7 +8,7 @@ We use an utility tool called mdBook to create online books from Markdown files.
 
 1. Install mdBook from [here](https://github.com/rust-lang/mdBook).
 2. Clone the repository by `git clone https://github.com/status-im/nimbus-eth2.git`.
-3. Go to where the Markdown files are located by `cd docs`.
+3. Go to where the Markdown files are located by `cd docs/the_nimbus_book/`.
 
 ## Real-Time Update and Preview Changes
 

docs/the_nimbus_book/src/eth1.md

@@ -1,17 +1,23 @@
 # Run an eth1 node
 
+> ⚠️ Warning: make sure you've copied the endpoint that starts with either `ws` or `wss` (websocket), and not `http` or `https`. Nimbus does not currently support `http` endpoints.
+
+
 In order to process incoming validator deposits from the eth1 chain, you'll need to run an eth1 client in parallel to your eth2 client. 
 
 Validators are responsible for including new deposits when they propose blocks. And an eth1 client is needed to ensure your validator performs this task correctly.
 
 On this page we provide instructions for using Geth (however, any reputable eth1 client should do the trick).
 
-> **Note:** If you're running on a resource-restricted device like a [Raspberry Pi](./pi-guide.md), we recommend [setting up a personal Infura endpoint](./infura-guide.md) instead as a stop-gap solution.
-> As it stands it may be a little complicated to run a full Geth node on a Pi (and light mode doesn't give you the deposit data you need).
->
->In the medium term (3-6 months), we expect someone (perhaps us) will build a thin layer on top of plain Eth1 header-syncing light clients to address this issue. Specifically, what's missing is a gossip network broadcasting deposit proofs (i.e. deposits and corresponding Merkle proofs rooted in Eth1 headers). When that happens, you should be able to swap out Infura.
->
-> However, if you have a > 500GB SSD, and your hardware can handle it, we strongly recommend running your own eth1 client. This will help ensure the network stays as decentralised as possible.
+> **Note:** If you have a > 500GB SSD, and your [hardware](./hardware.md) can handle it, we strongly recommend running your own eth1 client. This will help ensure the network stays as decentralised as possible. If you can't however, the next best option is to set up a 3rd part provider like [infura](./infura-guide.md).
+
+## Nimbus
+In parallel to `nimbus-eth2` we are working hard on our [our exectution client](https://github.com/status-im/nimbus-eth1). While this is  very much a project in development (i.e. not yet ready for public consumption), we welcome you to experiment with it.
+
+## Nethermind
+*TBC*
+
+## Geth
 
 ### 1. Install Geth
 If you're running MacOS, follow the instructions [listed here](https://github.com/ethereum/go-ethereum/wiki/Installation-Instructions-for-Mac) to install geth. Otherwise [see here](https://github.com/ethereum/go-ethereum/wiki/Installing-Geth).
@@ -37,7 +43,7 @@ geth --ws
 
 Let it sync - Geth uses a fast sync mode by default. It may take anywhere between a few hours and a couple of days.
 
->**N.B. It is safe to run Nimbus and start validating even if Geth hasn't fully synced yet.**
+>**N.B.** It is safe to run Nimbus and start validating even if Geth hasn't fully synced yet
 
 You'll know Geth has finished syncing, when you start seeing logs that look like the following:
 
@@ -54,5 +60,9 @@ INFO [05-29|01:16:14] Imported new chain segment               blocks=1 txs=11
 ```
 
 
+Geth accepts connections from the loopback interface (`127.0.0.1`), with default WebSocket port `8546`. This means that your default Web3 provider URL should be: `ws://127.0.0.1:8546`
+
+
+
 
 

docs/the_nimbus_book/src/health.md

@@ -0,0 +1,103 @@
+# Monitor the health of your node
+
+The most important thing for the the health, performance and stablity of your node and the overall network is the strength of your node's network connectivity / peer count.
+
+## Monitor your Peer count
+
+If your Peer count is low (less than `20`) and/or you repeatedly see the following warning:
+```
+WRN 2021-05-08 12:59:26.669+00:00 Peer count low, no new peers discovered    topics="networking" tid=1914 file=eth2_network.nim:963 discovered_nodes=9 new_peers=0 current_peers=1 wanted_peers=160
+```
+
+It probably means that your computer is not reachable from the outside. This means you won't be able to accept any incoming peer connections.
+
+If you're on a home network, the fix here is to set up port forwarding.
+
+## Set up port forwarding
+
+If you're running on a home network and want to ensure you are able to receive incoming connections you may need to set up port forwarding (though some routers automagically set this up for you).
+
+
+> **Note:** If you are running your node on a virtual public cloud (VPC) instance, you can safely ignore this section.
+
+While the specific steps required vary based on your router, they can be summarised as follows:
+
+
+1. Determine your [public IP address](./health.md#public-ip-address)
+2. Determine your [private IP address](./health.html#private-ip-address)
+3. Browse to the management website for your home router (typically [http://192.168.1.1)](http://192.168.1.1)
+4. Log in as admin / root
+5. Find the section to configure port forwarding
+6. Configure a port forwarding rule with the following values:
+- External port: `9000`
+- Internal port: `9000`
+- Protocol: `TCP`
+- IP Address: Private IP address of the computer running Nimbus
+7. Configure a second port forwarding rule with the following values:
+- External port: `9000`
+- Internal port: `9000`
+- Protocol: `UDP`
+- IP Address: Private IP address of the computer running Nimbus
+
+### Determine your public IP address
+
+To determine your public IP address, visit [http://v4.ident.me/](http://v4.ident.me/) or run this command:
+
+```
+curl v4.ident.me
+```
+
+### Determine your private IP address
+
+To determine your private IP address, run the appropriate command for your OS:
+
+**Linux:**
+
+```
+ip addr show | grep "inet " | grep -v 127.0.0.1
+```
+
+**Windows:**
+
+```
+ipconfig | findstr /i "IPv4 Address"
+```
+
+**macOS:**
+
+```
+ifconfig | grep "inet " | grep -v 127.0.0.1
+```
+
+
+### Check open ports on your connection
+Use [this tool](https://www.yougetsignal.com/tools/open-ports/) to check your external (public) IP address and detect open ports on your connection (Nimbus TCP and UDP ports are both set to `9000` by default).
+
+## Keep track of your attestation effectiveness
+
+Attestation effectiveness is a metric that directly affects your validator rewards. In simple terms, an attestation is more valuable the sooner it is put into a block and included in the chain. 
+
+This interval is called the *inclusion distance* of an attestation. The smaller it is, the more profitable your validator will be. For a deeper understanding we highly recommend reading [Attestant's wonderful blog post](https://www.attestant.io/posts/defining-attestation-effectiveness/#:~:text=Stakers%20looking%20to%20maximize%20their,provide%20clear%20metrics%20for%20performance.) on the matter.
+
+You can verify your validator's effectiveness on the [beaconcha.in](https://beaconcha.in/) website.
+
+![](https://i.imgur.com/u80Ub2j.png)
+
+Ideally you want to see a value above 80%.
+
+While attestation effectiveness depends on a variety of factors - attestation network propagation, your network connectivity, and the peers you are connected to - your network connectivity is likely the most important factors you can control to improve this metric. Apart from the tips outlined on this guide, you could also experiment with [subscribing to all subnets](./profits.md#subscribe-to-all-subnets).
+
+## Monitor your system's network I/O usage
+
+If you're a Linux user and want to track how much network I/O your system uses over time, you can install a nice utility called [`vnstat`](https://humdi.net/vnstat/).
+
+To install, run:
+
+```
+sudo apt install vnstat
+```
+
+To run it:
+
+*TBC -See [here](https://github.com/jclapis/rp-pi-guide/blob/main/Native.md#monitoring-your-pis-performance) for more*
+

docs/the_nimbus_book/src/infura-guide.md

@@ -5,7 +5,7 @@ In a nutshell, Infura is a hosted ethereum node cluster that lets you make reque
 
 While we do support Infura to process incoming validator deposits, we recommend running your own eth1 node to avoid relying on a third-party-service.
 
-> **Note:** Nimbus currently supports remote Infura nodes and [local Geth nodes](./eth1.md). In the future, we plan on having our own eth1 client -- [Nimbus 1](https://github.com/status-im/nimbus) -- be the recommended default.
+> **Note:** Nimbus currently supports remote Infura nodes and [local Geth nodes](./eth1.md#geth). In the future, we plan on having our own eth1 client -- [Nimbus 1](https://github.com/status-im/nimbus) -- be the recommended default.
 
 ### 1. Visit Infura.io
 

docs/the_nimbus_book/src/keys.md

@@ -16,12 +16,14 @@ build/nimbus_beacon_node deposits import --data-dir=build/data/shared_mainnet_0
 >
 >*Prater*
 >```
->build/nimbus_beacon_node deposits import  --data-dir=build/data/shared_prater_0 "<YOUR VALIDATOR KEYS DIRECTORY>"
+>build/nimbus_beacon_node deposits import \
+> --data-dir=build/data/shared_prater_0 "<YOUR VALIDATOR KEYS DIRECTORY>"
 > ```
 > 
 > *Mainnet*
 > ```
->build/nimbus_beacon_node deposits import  --data-dir=build/data/shared_mainnet_0 "<YOUR VALIDATOR KEYS DIRECTORY>"
+>build/nimbus_beacon_node deposits import \ 
+> --data-dir=build/data/shared_mainnet_0 "<YOUR VALIDATOR KEYS DIRECTORY>"
 >```
 > 
 > Replacing `<YOUR VALIDATOR KEYS DIRECTORY>` with the full pathname of the `validator_keys` directory that was created when you generated your keys using the [command line app](https://github.com/ethereum/eth2.0-deposit-cli/releases/).

docs/the_nimbus_book/src/migration.md

@@ -36,7 +36,9 @@ It's important that you disable the Prysm validator as well as stopping it, to p
 Run the following to export your Prysm validator's [slashing protection](https://eips.ethereum.org/EIPS/eip-3076) history:
 
 ```
-prysm.sh validator slashing-protection export --datadir=/your/prysm/wallet --slashing-protection-export-dir=/path/to/export_dir
+prysm.sh validator slashing-protection export \ 
+ --datadir=/your/prysm/wallet \ 
+ --slashing-protection-export-dir=/path/to/export_dir
 ```
 
 To be extra sure that your validator has stopped, wait a few epochs and confirm that your validator have stopped attesting (check [beaconcha.in](https://beaconcha.in/)).

docs/the_nimbus_book/src/more-keys.md

@@ -20,12 +20,18 @@ Run the following command from the directory which contains the `deposit` execut
 
 **Prater**
 ```
-./deposit existing-mnemonic --validator_start_index 0 --num_validators 1 --chain prater
+./deposit existing-mnemonic \
+ --validator_start_index 0 \
+ --num_validators 1 \
+ --chain prater
 ```
 
 **Mainnet**
 ```
-./deposit existing-mnemonic --validator_start_index 0 --num_validators 1 --chain mainnet
+./deposit existing-mnemonic \
+ --validator_start_index 0 \
+ --num_validators 1 \
+ --chain mainnet
 ```
 
 You'll be prompted to enter your mnemonic, and a new password for your keystore.
@@ -46,12 +52,18 @@ Run the following command from the directory which contains the `deposit` execut
 
 **Prater**
 ```
-./deposit existing-mnemonic --validator_start_index 1 --num_validators 1 --chain prater
+./deposit existing-mnemonic \
+ --validator_start_index 1 \
+  --num_validators 1 \
+  --chain prater
 ```
 
 **Mainnet**
 ```
-./deposit existing-mnemonic --validator_start_index 1 --num_validators 1 --chain mainnet
+./deposit existing-mnemonic \
+ --validator_start_index 1 \
+ --num_validators 1 \
+ --chain mainnet
 ```
 
 You'll be prompted to enter your mnemonic, and a new password for your keystore.

docs/the_nimbus_book/src/options.md

@@ -1,8 +1,6 @@
 # Command line options
 
-> ⚠️  If you want to add your own options, remember that the format is `--foo=bar`, `--baz`, or `--foo-bar=qux`
-
-You can pass any `nimbus_beacon_node` options to the `prater` and `mainnet` scripts. For example, if you wanted to launch Nimbus on mainnet with different base ports than the default `9000/udp` and `9000/tcp`, say `9100/udp` and `9100/tcp`, you would run:
+You can pass any `nimbus_beacon_node` options to the `prater` and `mainnet` scripts. For example, if you want to launch Nimbus on mainnet with different base ports than the default `9000/udp` and `9000/tcp`, say `9100/udp` and `9100/tcp`, run:
 
 ```
 ./run-mainnet-beacon-node.sh --tcp-port=9100 --udp-port=9100

docs/the_nimbus_book/src/pi-guide.md

@@ -295,7 +295,7 @@ build/nimbus_beacon_node deposits import  --data-dir=build/data/shared_pyrmont_0
 
 We're finally ready to connect to the Prater testnet!
 
->**Note:** If you haven't already, we recommend registering for, and running, your own Infura endpoint to connect to eth1. For instruction on how to do so, see [this page](./infura-guide.md).
+>**Note:** If you haven't already, we recommend registering for, and running, your own eth1 node in parallel. For instruction on how to do so, see [this page](./eth1.md).
 
 To connect to Prater, run:
 ```

docs/the_nimbus_book/src/profits.md

@@ -0,0 +1,52 @@
+# Optimise for profitability
+
+Profitability depends heavily on the network and peer quality.
+
+While block proposals are more lucrative than attestations, they are much rarer.
+
+
+## Check for next action before restarting
+
+
+To see when your validator is next due to make an attestation or proposal pay attention to the `Slot end` messages in your logs:
+```
+INF 2021-05-31 17:46:11.094+02:00 Slot end
+topics="beacnde" tid=213670 file=nimbus_beacon_node.nim:932
+slot=1304329
+nextSlot=1304330
+head=cffee454:38460
+headEpoch=1201
+finalizedHead=077da232:38368
+finalizedEpoch=1199
+nextAttestationSlot=338638
+nextProposalSlot=-1
+nextActionWait=4m35s874ms405us837ns
+```
+
+Specifically, have a look at `nextActionWait` time.
+
+
+If you're concerned about missing an attestation or proposal, wait until `nextActionWait` is greater than 4 minutes or so before restarting Nimbus.
+
+
+You can also use the `nimbus-eth2` [API](./api.md). For example, to check if your validator has a next Proposal slot assigned, run:
+
+```
+curl -d '{"jsonrpc":"2.0","method":"get_v1_validator_duties_proposer","params":[${EPOCH_NUMBER_OF_INTEREST}],"id":1}' -H 'Content-Type: application/json' localhost:9190 -s | jq ".result[]" | grep ${PATTERN_WHICH_MATCHES_VALIDATOR_PUBLIC_KEYS}
+```
+
+
+## Subscribe to all subnets
+Launching the beacon node with the `--subscribe-all-subnets` option increases bandwidth and cpu usage, but helps the network and makes the block production algorithm perform slightly better.
+
+To elaborate a little, without this option enabled Nimbus only listens to a subset of the attestation traffic - in particular, Nimbus doesn't listen to all unaggregated traffic but instead relies on peers to aggregate attestations on the subnets it doesn't subscribe to. 
+
+With this option enabled, Nimbus listens to all unaggregated channels (subscribes to all subnets). Practically speaking, this means that when producing a block, Nimbus can "top up" the aggregates that other peers have made with it's own unaggregated attestations. This can lead to better packing in some cases, which can lead to slightly greater rewards.
+
+
+
+
+
+
+
+

docs/the_nimbus_book/src/start-syncing.md

@@ -5,27 +5,25 @@ If you're joining a network that has already launched, you need to ensure that y
 
 This is particularly important if you are joining a network that's been running for a while.
 
+> **N.B.** In order to process incoming validator deposits from the eth1 chain, you'll need to run an eth1 client (**web3 provider**) in parallel to your eth2 client. See [here](./eth1.md) for instructions on how to do so.
+
 ### Testnet
 
 To start syncing the `prater` testnet , from the `nimbus-eth2` repository, run:
 
 ```
- ./run-prater-beacon-node.sh 
+ ./run-prater-beacon-node.sh --web3-url="<YOUR_WEB3_PROVIDER_URL>"
 ```
 
 ### Mainnet
 
 
-To start syncing the eth2 mainnet, while monitoring the eth1 mainnet chain for deposits, run:
+To start syncing the eth2 mainnet, run:
 
 ```
- ./run-mainnet-beacon-node.sh --web3-url="ws://127.0.0.1:8546"
+ ./run-mainnet-beacon-node.sh --web3-url="<YOUR_WEB3_PROVIDER_URL>"
 ```
 
-Note, the above command assumes you are running a [local geth instance](./eth1.md). Geth accepts connections from the loopback interface (`127.0.0.1`), with default WebSocket port `8546`. This means that your default Web3 provider URL should be: `ws://127.0.0.1:8546`
-
->**N.B.** If you're using [your own Infura endpoint](./infura-guide), you should enter that instead.
-
 You should see the following output:
 
 ```

docs/the_nimbus_book/src/troubleshooting.md

@@ -52,8 +52,8 @@ If you're experiencing a low peer count, you may be behind a firewall. Try resta
 ```
 ### noCommand does not accept arguments
 
-If, on start,  you see
-```The command 'noCommand' does not accept arguments```
+If, on start,  you see `The command 'noCommand' does not accept arguments`
+
 Double check to see if your command line flags are in the correct format, i.e. `--foo=bar`, `--baz`, or `--foo-bar=qux`.
 
 ### Address already in use error
@@ -105,6 +105,24 @@ If you see an error that looks like the following:
 
 It's because your node can't connect to the web3 provider you have specified. Please double check that you've correctly specified your provider. If you haven't done so already, we recommend [adding a backup](web3-backup.md).
 
+### Discovered new external address warning log
+
+```console
+WRN 2021-03-11 13:26:25.943-08:00
+Discovered new external address but ENR auto update is off
+topics="discv5" tid=77655 file=protocol.nim:940 majority=Some("myIPaddressHere":9000) previous=None[Address]
+```
+
+This message is displayed regularly when Nimbus canot detect your correct IP address. It may be a sign that you have a dynamic IP address that keeps changing. Or that Nimbus is unable to get your IP from the [UPnP](https://en.wikipedia.org/wiki/Universal_Plug_and_Play).
+
+The first step is to try relaunching the beacon node with the `--enr-auto-update` option.
+
+If that doesn't fix the problem, double check that your [ports are open](https://www.yougetsignal.com/tools/open-ports/) and that you have [port forwarding](https://www.computerhope.com/issues/ch001201.htm) enabled on your gateway (assuming that you are behind a [NAT](https://en.wikipedia.org/wiki/Network_address_translation)).
+
+See our page on [monitoring the health of your node](./health.md) for more.
+
+
+
 ## Raspberry Pi
 
 ### Trouble transferring data to/from USB3.0 SSDs

docs/the_nimbus_book/src/voluntary-exit.md

@@ -2,26 +2,31 @@
 
 > ⚠️  Voluntary exits are **irreversible**. You won't be able to validate again with the same key. And you won't be able to withdraw your stake until the Eth1 and Eth2 merge. *Note that voluntary exits won't be processed if the chain isn't finalising.*
 
-To perform a voluntary exit, with your beacon node running, run:
+To perform a voluntary exit, make sure your beacon node is running with the `--rpc`option enabled (e.g. `./run-mainnet-beacon-node.sh --rpc`), then run:
 
 **Pyrmont**
 
 ```
-build/nimbus_beacon_node deposits exit --validator=<VALIDATOR_PUBLIC_KEY> --data-dir=build/data/shared_pyrmont_0
+build/nimbus_beacon_node deposits exit \
+ --validator=<VALIDATOR_PUBLIC_KEY> \
+ --data-dir=build/data/shared_pyrmont_0
 ```
 
 **Prater**
 
 ```
-build/nimbus_beacon_node deposits exit --validator=<VALIDATOR_PUBLIC_KEY> --data-dir=build/data/shared_prater_0
-
+build/nimbus_beacon_node deposits exit \ 
+ --validator=<VALIDATOR_PUBLIC_KEY> \ 
+ --data-dir=build/data/shared_prater_0
 ```
 
 
 **Mainnet**
 
 ```
-build/nimbus_beacon_node deposits exit --validator=<VALIDATOR_PUBLIC_KEY> --data-dir=build/data/shared_mainnet_0
+build/nimbus_beacon_node deposits exit \ 
+ --validator=<VALIDATOR_PUBLIC_KEY> \
+ --data-dir=build/data/shared_mainnet_0
 ```
 
 > **Note:** Make sure your `<VALIDATOR_PUBLIC_KEY>` is prefixed with `0x`. In other words the public key should look like `0x95e3...`

docs/the_nimbus_book/src/web3-backup.md

@@ -2,9 +2,11 @@
 
 It's a good idea to add a backup web3 provider in case your main one goes down. You can do this by simply repeating the `--web3-url` parameter on launch.
 
-For example, if your primary eth1 node is a [local Geth](./eth1.md), but you want to use [Infura](./infura-guide.md) as a backup you would run:
+For example, if your primary eth1 node is a [local Geth](./eth1.md#geth), but you want to use [Infura](./infura-guide.md) as a backup you would run:
 
 ```
-./run-mainnet-beacon-node.sh  --web3-url="ws://127.0.0.1:8546" --web3-url="wss://mainnet.infura.io/ws/v3/..."
+./run-mainnet-beacon-node.sh \
+ --web3-url="ws://127.0.0.1:8546" \
+ --web3-url="wss://mainnet.infura.io/ws/v3/..."
 ```