status-im
/
infra-nimbus
mirror of https://github.com/status-im/infra-nimbus.git
2
0
Fork 0
Code Issues Projects Releases Wiki Activity
879 Commits 10 Branches 0 Tags 31 MiB
Commit Graph

420 Commits

Author SHA1 Message Date
Jakub Sokołowski 5344827479
nimbus.prater: deploy Nethermind node on linux-02 
Part of work to use Nethermind for eth2.prod fleet:
https://github.com/status-im/infra-eth2/issues/11

Signed-off-by: Jakub Sokołowski <jakub@status.im>
 2023-08-07 23:05:33 +02:00
Jakub Sokołowski 5586db729d
nimbus.prater: reduce max_headers_size to 128 KB 
Probably the reason for elevated memory usage when using Validator
Client with large number of validators attached.

Signed-off-by: Jakub Sokołowski <jakub@status.im>
 2023-08-07 14:54:07 +02:00
Jakub Sokołowski 2c9bbe832b
nimbus.sepolia: open Geth Websocket ports for Vac 
Requested by p1ge0nh8er for vacdev.misc host.

Signed-off-by: Jakub Sokołowski <jakub@status.im>
 2023-07-16 10:57:40 +02:00
Jakub Sokołowski f304db1cc0
nimbus.prater: bump Eirgon memory limit to 15% 
Signed-off-by: Jakub Sokołowski <jakub@status.im>
 2023-07-16 10:18:40 +02:00
Jakub Sokołowski 41025265e2
nimbus.prater: add 16 GB SWAP file, no SWAP partition 
There were OOM killer logs on `linux-06` due to Erigon.

Signed-off-by: Jakub Sokołowski <jakub@status.im>
 2023-07-12 09:38:19 +02:00
Jakub Sokołowski 7dff81fb2f
add windows-01.ih-eu-mda1.nimbus.prater host 
https://github.com/status-im/infra-nimbus/issues/132

Signed-off-by: Jakub Sokołowski <jakub@status.im>
 2023-07-07 12:17:33 +02:00
Jakub Sokołowski 4480d292be
nimbus.sepolia: debug flag for old attestation stability 
As requested by Dustin.

Signed-off-by: Jakub Sokołowski <jakub@status.im>
 2023-07-06 08:32:55 +02:00
Jakub Sokołowski cec778f4f1
nimbus.prater: debug flag for old attestation stability 
As requested by Dustin.

Signed-off-by: Jakub Sokołowski <jakub@status.im>
 2023-07-06 08:32:54 +02:00
Jakub Sokołowski 52e518d3c3
nimbus.sepolia: drop nim_commit=version-1-6 flag 
It no longer has any effect.

Signed-off-by: Jakub Sokołowski <jakub@status.im>
 2023-07-06 08:32:53 +02:00
Jakub Sokołowski 9455a6a5ae
nimbus.prater: drop nim_commit=version-1-6 flag 
It no longer has any effect.

Signed-off-by: Jakub Sokołowski <jakub@status.im>
 2023-07-06 08:29:09 +02:00
Jakub Sokołowski 44d77c54f4
nimbus.prater: move validators from Hetzner hosts 
The InnovaHosting servers are almost fully synced, time to move.
https://github.com/status-im/infra-nimbus/issues/132

Signed-off-by: Jakub Sokołowski <jakub@status.im>
 2023-07-06 08:29:08 +02:00
kdeme e0140a1497
nimbus.fluffy: drop bootstrap nodes, use network flag 
The `testnet0` network used with `--network` flag already includes our
own Fluffy nodes from this fleet, so no need to define them separately.

Signed-off-by: Jakub Sokołowski <jakub@status.im>
 2023-06-29 18:23:53 +02:00
Jakub Sokołowski f24fa170ee
add metal-01.ih-eu-mda1.nimbus.eth1 host 
https://github.com/status-im/infra-nimbus/issues/132

Signed-off-by: Jakub Sokołowski <jakub@status.im>
 2023-06-26 21:59:13 +02:00
Jakub Sokołowski ee7675543a
nimbus-geth-goerli: fix paths for extra data volume 
Otherwise `/docker` is part of root and there's not enough space.

Signed-off-by: Jakub Sokołowski <jakub@status.im>
 2023-06-26 12:54:54 +02:00
Jakub Sokołowski 5b85f5c26d
add three InnovaHosting servers to nimbus.geth fleet 
https://github.com/status-im/infra-nimbus/issues/132

Signed-off-by: Jakub Sokołowski <jakub@status.im>
 2023-06-23 21:15:19 +02:00
Jakub Sokołowski de28d46133
nimbus.mainnet: enable yamux for 2 libp2p nodes 
https://github.com/status-im/infra-nimbus/issues/148

Signed-off-by: Jakub Sokołowski <jakub@status.im>
 2023-06-23 15:14:42 +02:00
Jakub Sokołowski 052a8485aa
nimubs.prater: add 4 new hosts from InnovaHosting 
https://github.com/status-im/infra-nimbus/issues/132

Signed-off-by: Jakub Sokołowski <jakub@status.im>
 2023-06-22 20:44:31 +02:00
Jakub Sokołowski 766092f537
nimbus.prater: fix validator monitor details condition 
It was causing very high cardinality due to lots of validators.
Crashed our Hetzner metrics server.

Signed-off-by: Jakub Sokołowski <jakub@status.im>
 2023-06-22 12:07:16 +02:00
Jakub Sokołowski 7d9ebb1744
nimbus.prater: double beacon node open files limit 
Causes issues with metrics responses due to inability to open more files.

Signed-off-by: Jakub Sokołowski <jakub@status.im>
 2023-06-22 09:31:22 +02:00
Jakub Sokołowski 5f151fb828
all: grant admin rights to cheatfate user 
Signed-off-by: Jakub Sokołowski <jakub@status.im>
 2023-06-21 16:50:17 +02:00
Jakub Sokołowski 6f0ff4ad19
nimbus.prater: run only two nodes on MacOS host 
Not enough disk space for 3 nodes.

Signed-off-by: Jakub Sokołowski <jakub@status.im>
 2023-06-20 19:47:20 +02:00
Jakub Sokołowski 1aa507c359
prater: move validators to new linux-01 and linux-02 
https://github.com/status-im/infra-nimbus/issues/132

Signed-off-by: Jakub Sokołowski <jakub@status.im>
 2023-06-16 12:57:24 +02:00
Jakub Sokołowski 43b238c87e
nimbus.all: use new --validator-monitor-details flag 
Replaced `--validator-monitor-totals` in:
https://github.com/status-im/nimbus-eth2/pull/4468

Signed-off-by: Jakub Sokołowski <jakub@status.im>
 2023-06-07 19:04:37 +02:00
Etan Kissling b33a35c5e1
nimbus: set --rest-allow-origin='*' for public_api 
To allow browser based clients to access our API servers.

Depends on:

- https://github.com/status-im/infra-role-beacon-node-linux/pull/14
- https://github.com/status-im/infra-role-beacon-node-linux/pull/15
- https://github.com/status-im/infra-role-beacon-node-macos/pull/7
- https://github.com/status-im/infra-role-beacon-node-macos/pull/8
- https://github.com/status-im/infra-role-beacon-node-windows/pull/6
- https://github.com/status-im/infra-role-beacon-node-windows/pull/7

Signed-off-by: Etan Kissling <etan@status.im>
Signed-off-by: Jakub Sokołowski <jakub@status.im>
 2023-06-05 13:45:55 +02:00
Jakub Sokołowski e08586be04
nimbus.sepolia: move validators to new Innova host 
https://github.com/status-im/infra-nimbus/issues/132

Signed-off-by: Jakub Sokołowski <jakub@status.im>
 2023-06-05 12:08:01 +02:00
Jakub Sokołowski 6ae7dc00f0
nimbus.prater: add Innova host config 
https://github.com/status-im/infra-nimbus/issues/132

Signed-off-by: Jakub Sokołowski <jakub@status.im>
 2023-06-02 10:35:18 +02:00
Jakub Sokołowski da40f535ba
nimbus.sepolia: add Innova host config 
https://github.com/status-im/infra-nimbus/issues/132

Signed-off-by: Jakub Sokołowski <jakub@status.im>
 2023-06-02 10:34:20 +02:00
Jakub Sokołowski 948b5885fe
nimbus.prater: move disabling discovery to Hetzner group 
Signed-off-by: Jakub Sokołowski <jakub@status.im>
 2023-05-29 21:20:03 +02:00
Jakub Sokołowski e9a05d73ab
nimbus.prater: add DNS discovery for Erigon 
Necessary because we disable normal discovery.

Source:
https://github.com/ethereum/discv4-dns-lists

Signed-off-by: Jakub Sokołowski <jakub@status.im>
 2023-05-29 21:03:53 +02:00
Jakub Sokołowski a8741b7019
nimbus.prater: , disable discovery for Erigon 
It appears peer discovery can be detected by hetzner as an attempt at
network scanning:
https://github.com/ledgerwatch/erigon/issues/7234

Which can lead to the server being shut down.
For this reason I'm lowering max peers and disabling discovery.

Signed-off-by: Jakub Sokołowski <jakub@status.im>
 2023-05-29 20:06:58 +02:00
Jakub Sokołowski 3f3d763990
nimbus.prater: double nofile limit to 32768 
Which needs to be more than 4 x 7899 validators.

Signed-off-by: Jakub Sokołowski <jakub@status.im>
 2023-05-29 18:36:09 +02:00
Jakub Sokołowski e5324beaaa
nimbus-geth-goerli: remove one of three Geth nodes 
Not nough storage space to run 3 nodes on the hosts.

Also made AWS, Windows and MacOS nodes discover available
Geth nodes using Consul catalog.

Signed-off-by: Jakub Sokołowski <jakub@status.im>
 2023-05-24 18:34:03 +02:00
Jakub Sokołowski 597b85a0c8
nimbus.sepolia: open EL WebSocket to wakuv2.test 
https://github.com/waku-org/nwaku/issues/1624
https://github.com/status-im/infra-nim-waku/pull/69

Signed-off-by: Jakub Sokołowski <jakub@status.im>
 2023-05-17 18:15:47 +02:00
Jakub Sokołowski 2e14ecc79e
nimbus.sepolia: enable SMART metrics service 
Signed-off-by: Jakub Sokołowski <jakub@status.im>
 2023-05-17 18:11:27 +02:00
Jakub Sokołowski c15e1ea358
nimbus.mainnet: 10k max peers for stable-02@linux-03 
Signed-off-by: Jakub Sokołowski <jakub@status.im>
 2023-05-16 12:38:24 +02:00
Jakub Sokołowski a270f4044c
all: get Consul certificates from BitWarden 
Signed-off-by: Jakub Sokołowski <jakub@status.im>
 2023-05-16 12:14:38 +02:00
Jakub Sokołowski 0cd6de1ea6
nimbus.mainnet: use ncli_db from stable node 
Signed-off-by: Jakub Sokołowski <jakub@status.im>
 2023-05-16 12:14:27 +02:00
Jakub Sokołowski d04a98ea11
remove metal-01.he-eu-hel1.nimbus.mainnet 
https://github.com/status-im/infra-nimbus/issues/132

Signed-off-by: Jakub Sokołowski <jakub@status.im>
 2023-05-16 11:05:30 +02:00
Jakub Sokołowski a145963a2a
ih-eu-mda1: disable Consul check for SMART metrics 
The service fails on InnovaHosting servers due to HBA controller:
https://github.com/prometheus-community/smartctl_exporter/issues/26

Signed-off-by: Jakub Sokołowski <jakub@status.im>
 2023-05-15 19:51:41 +02:00
Jakub Sokołowski 0256a21dd0
nimbus.mainnet: add two more hosts from Innova 
The `linux-06` is a new host, while `linux-07` is re-purposed `linux-01`
from the prater fleet. This is done because the Innova company is out of
Xeon E5-2690 CPUs, and we want consistent performance across the fleet.

This way all seven hosts in `nimbus.mainnet` will use the E5-2690 CPU.

https://github.com/status-im/infra-nimbus/issues/132

Signed-off-by: Jakub Sokołowski <jakub@status.im>
 2023-05-09 15:07:22 +02:00
Jakub Sokołowski 406ac58f16
nimbus.mainnet: use --no-el for even number nodes 
Attempted fix issues with syncing where latency on the Geth endpoint is
too high for beacon nodes to finish syncing properly.

Signed-off-by: Jakub Sokołowski <jakub@status.im>
 2023-05-05 20:58:53 +02:00
Jakub Sokołowski 01cf22c9f8
get-geth-api-urls: rename variables for clarity 
Signed-off-by: Jakub Sokołowski <jakub@status.im>
 2023-05-05 19:21:49 +02:00
Jakub Sokołowski 5635b39e50
mainnet: decommission 6 Hetzner hosts 
Being replaced by InnovaHosting servers
https://github.com/status-im/infra-nimbus/issues/132

I'm leaving old `metal-07` as `metal-01` since we don't have a server
with `libp2p` nodes on mainnet yet.

Signed-off-by: Jakub Sokołowski <jakub@status.im>
 2023-04-28 12:22:46 +02:00
Jakub Sokołowski cd0841eba2
ih-eu-mda1: fix destintion DC for sending logs 
Signed-off-by: Jakub Sokołowski <jakub@status.im>
 2023-04-25 18:06:00 +02:00
Jakub Sokołowski e43ede93a5
nimbus.mainnet: fix exec layer URL logic 
Signed-off-by: Jakub Sokołowski <jakub@status.im>
 2023-04-20 22:28:26 +02:00
Jakub Sokołowski 940b2577da
nimbus.mainnet: add node layout for new Innova hosts 
Once they are synced and working we'll remove the Hetzner ones.

https://github.com/status-im/infra-nimbus/issues/132

Signed-off-by: Jakub Sokołowski <jakub@status.im>
 2023-04-18 18:14:53 +02:00
Jakub Sokołowski 8ddbd86139
nimbus.mainnet: revert removing of stable-02 node 
This reverts commit ab266d132d.

Signed-off-by: Jakub Sokołowski <jakub@status.im>
 2023-04-17 20:06:57 +02:00
Jakub Sokołowski 53ccd34927
nimbus.prater: bootstrap new Innova servers 
https://github.com/status-im/infra-nimbus/issues/132

Signed-off-by: Jakub Sokołowski <jakub@status.im>
 2023-04-17 17:11:27 +02:00
Jakub Sokołowski 1f5ad05893
nimbus.fluffy: 24h to make healthcheck critical 
This is not a critical service and there's a LOT of sevices that cause a
lot of alerts when developers are stopping and starting them.

Signed-off-by: Jakub Sokołowski <jakub@status.im>
 2023-04-12 19:28:37 +02:00
Jakub Sokołowski 9ec35f149b
nimbus.fluff: deploy smart-metrics exporter 
Signed-off-by: Jakub Sokołowski <jakub@status.im>
 2023-04-12 13:24:29 +02:00