status-im
/
infra-nimbus
mirror of https://github.com/status-im/infra-nimbus.git
2
0
Fork 0
Code Issues Projects Releases Wiki Activity
1,175 Commits 10 Branches 0 Tags 31 MiB
Commit Graph

560 Commits

Author SHA1 Message Date
Jakub Sokołowski 24020d0962
all: reduce MTR report cycle from 10 to 1 
We have received a complaint from InnovaHosting about them being hit by
about 150 ICMP `ttl1` packets/s on their routers, causing excess CPU usage.
https://client.innovahosting.net/viewticket.php?tid=532874&c=8gALx9vm

By using `tcpdump` I have identified that `mtr` by default pings the
target 10 times, which means that the default value of `-c`/`--report-cycles`
is 10, although this is not documented in the manual.

We can see this when calling `mtr github.com` and watching with `tcpdump`:
```
 > sudo tcpdump -v -i eno1 icmp and src 185.181.230.78 and dst github.com | grep 'ttl 1,'
tcpdump: listening on eno1, link-type EN10MB (Ethernet), snapshot length 262144 bytes
19:54:53.981243 IP (tos 0x0, ttl 1, id 37119, offset 0, flags [none], proto ICMP (1), length 64)
...(8 packets omitted)...
19:55:03.025460 IP (tos 0x0, ttl 1, id 38226, offset 0, flags [none], proto ICMP (1), length 64)
```
We don't need to run the test 10 times to get a result for our metric.

Related to:
https://github.com/status-im/infra-role-bootstrap-linux/commit/ea22bdfe

Signed-off-by: Jakub Sokołowski <jakub@status.im>
 2023-11-20 21:07:17 +01:00
Jakub Sokołowski 6b800a5342
nimbus.fluffy: re-enable Consul healthchecks 
It appears the RPC issues was resolved in:
https://github.com/status-im/nimbus-eth1/issues/1880

Most probably caused by DB size.

Signed-off-by: Jakub Sokołowski <jakub@status.im>
 2023-11-20 09:57:46 +01:00
Jakub Sokołowski 2cac3081a0
layouts: add script and generate TSVs of validators 
Helps developers identify which host holds which validator.

Signed-off-by: Jakub Sokołowski <jakub@status.im>
 2023-11-17 15:01:57 +01:00
Jakub Sokołowski c29b23c6dc
nimbus.sepolia: open ports for waku.test fleet 
Signed-off-by: Jakub Sokołowski <jakub@status.im>
 2023-11-14 11:56:16 +01:00
kdeme ce37186651
all: update SSH key for kim 
This one is from a YubiKey.

Signed-off-by: Jakub Sokołowski <jakub@status.im>
 2023-11-07 18:48:05 +01:00
Jakub Sokołowski 10dd722e29
all: grant admin rights to kim 
Necessary to run 'perf'.

Signed-off-by: Jakub Sokołowski <jakub@status.im>
 2023-11-07 16:19:24 +01:00
Jakub Sokołowski c1be589960
all: add debug tools like gdb and perf 
Also allow use of 'perf' without root.

Signed-off-by: Jakub Sokołowski <jakub@status.im>
 2023-11-07 12:48:05 +01:00
Jakub Sokołowski 4df34ac3c1
nimbus.sepolia: enable payload builder for 4th node 
Also drop unnecessary Nim build flags.

Signed-off-by: Jakub Sokołowski <jakub@status.im>
 2023-11-07 11:50:26 +01:00
Jakub Sokołowski 36f78a5970
nimbus.fluffy: disable Consul healthchecks 
They are too flaky to be useful, see:
https://github.com/status-im/nimbus-eth1/issues/1880

Signed-off-by: Jakub Sokołowski <jakub@status.im>
 2023-11-02 12:51:01 +01:00
Jakub Sokołowski d2feb628c4
nimbus.fluffy: raise Consul alert threshold limits 
This host has constantly issue with nodes and nobody cares.

Signed-off-by: Jakub Sokołowski <jakub@status.im>
 2023-11-02 10:51:57 +01:00
Jakub Sokołowski 7272d55105
nimbus.prater: drop chronos and erigon from linux-06 
The host was overloaded and ran out of disk space on `/docker` volume.

Signed-off-by: Jakub Sokołowski <jakub@status.im>
 2023-11-02 10:25:40 +01:00
Jakub Sokołowski a6dc16830d
all: grant SSH access to ujscale, mumar@status 
Necessary to look at full Nimbus Prater logs.

Signed-off-by: Jakub Sokołowski <jakub@status.im>
 2023-10-31 14:04:26 +01:00
Jakub Sokołowski bd9d7cc752
all: prevent SPAM Nimbus logs from reaching Logstash 
Depends on:
https://github.com/status-im/infra-role-bootstrap-linux/commit/20609731
https://github.com/status-im/infra-role-bootstrap-linux/commit/98816e2a

Signed-off-by: Jakub Sokołowski <jakub@status.im>
 2023-10-24 16:13:01 +02:00
Jakub Sokołowski a86a65c4bc
nimbus.prater: disable log aggregation for the fleet 
Zahary agreed that we need to start phasing out use of Prater.
This also helps us avoid paying extra for 10 Gbps link for aggr host.

Signed-off-by: Jakub Sokołowski <jakub@status.im>
 2023-10-23 13:02:39 +02:00
Jakub Sokołowski d2270feece
logs.nimbus: increase total_fields.limit to 1500 
This is a special case since these logs are all custom JSON, so
increasing this is fine for now. I can't control what they put in logs.
https://discuss.elastic.co/t/approaches-to-deal-with-limit-of-total-fields-1000-in-index-has-been-exceeded/241039

Signed-off-by: Jakub Sokołowski <jakub@status.im>
 2023-10-12 18:56:21 +02:00
Jakub Sokołowski 63de71f759
all: remove SSH access for tanguy 
Signed-off-by: Jakub Sokołowski <jakub@status.im>
 2023-10-12 18:00:59 +02:00
Jakub Sokołowski 6855fc016b
ih-eu-mda1: drop data center override 
We now have Consul, logs, and metrics hosts in `ih-eu-mda1`.
https://github.com/status-im/infra-hq/issues/105

Signed-off-by: Jakub Sokołowski <jakub@status.im>
 2023-10-12 17:26:08 +02:00
Jakub Sokołowski 874771e109
nimbus.prater: fix port clash between Erigon and Geth 
Signed-off-by: Jakub Sokołowski <jakub@status.im>
 2023-10-09 11:15:32 +02:00
Jakub Sokołowski c6fc550cf3
nimbus.holesky: fix Erigon DevP2P port offset 
It needs to be 10, because a special flag `--p2p.allowed-ports` is used
to open multiple ports for multiple enabled Eth protocol versions.

For more information you can see:
https://github.com/status-im/infra-role-erigon/commit/eaef1e9f
https://github.com/ledgerwatch/erigon/issues/8330

Signed-off-by: Jakub Sokołowski <jakub@status.im>
 2023-10-02 19:42:16 +02:00
Alexis Pentori 458652e7f8
sepolia: Exposing ERA files 
Signed-off-by: Alexis Pentori <alexis@status.im>
 2023-10-02 13:42:51 +02:00
Jakub Sokołowski 5e12025aa6
all: grant admin ot Dustin user 
Signed-off-by: Jakub Sokołowski <jakub@status.im>
 2023-09-29 10:52:36 +02:00
Jakub Sokołowski adc1a061c4
nimbus.holesky: use the same ports for all EL node types 
Otherwise we'd need some kind of weird logic to compile the list of URLs
used by the beacon node, and the node types are exclusive so this is fine.

Signed-off-by: Jakub Sokołowski <jakub@status.im>
 2023-09-28 14:43:28 +02:00
Jakub Sokołowski 7b45d24b43
nimbus.holesky: upgrade Geth to 1.13.2 
Drop usage of master build.

Signed-off-by: Jakub Sokołowski <jakub@status.im>
 2023-09-28 11:54:00 +02:00
Jakub Sokołowski 2588a658cf
nimbus.holesky: use 2.49.3 Erigon release 
https://github.com/ledgerwatch/erigon/releases/tag/v2.49.3

Signed-off-by: Jakub Sokołowski <jakub@status.im>
 2023-09-27 19:54:35 +02:00
Jakub Sokołowski 7db4374fc9
nimbus.holesky: drop index from BN and VC names 
Signed-off-by: Jakub Sokołowski <jakub@status.im>
 2023-09-27 19:49:36 +02:00
Jakub Sokołowski ebb9cc82b3
nimbus.holesky: upgrade EL nodes to support new genesis 
https://github.com/status-im/infra-nimbus/issues/152

Signed-off-by: Jakub Sokołowski <jakub@status.im>
 2023-09-27 19:49:35 +02:00
Jakub Sokołowski 5446b3fc0f
nimbus.holesky: open metrics ports for EL nodes 
https://github.com/status-im/infra-nimbus/issues/152

Signed-off-by: Jakub Sokołowski <jakub@status.im>
 2023-09-16 11:38:40 +02:00
Jakub Sokołowski f200a1b4c5
nimbus.holesky: fleet config and validator layout 
https://github.com/status-im/infra-nimbus/issues/152

Signed-off-by: Jakub Sokołowski <jakub@status.im>
 2023-09-15 16:25:38 +02:00
Alexis Pentori 73184446d6
nimbus.parter: removing ephemeral debug option 
Signed-off-by: Alexis Pentori <alexis@status.im>
 2023-09-15 14:37:39 +02:00
Alexis Pentori e8834f4a9e
nimbus.sepolia: removing ephemeral debug option 
Signed-off-by: Alexis Pentori <alexis@status.im>
 2023-09-15 14:37:39 +02:00
Daniil Sobol b9373b7889
all: grant SSH access to daniil@status.im 
Signed-off-by: Jakub Sokołowski <jakub@status.im>
 2023-09-11 09:41:05 +02:00
Jakub Sokołowski a658d312a8
nimbus.prater: add stable node to macm1-01 host 
https://github.com/status-im/infra-nimbus/issues/132

Signed-off-by: Jakub Sokołowski <jakub@status.im>
 2023-09-06 10:19:21 +02:00
Jakub Sokołowski 227206c82d
nimbus.prater: move validators to macm1-01 
https://github.com/status-im/infra-nimbus/issues/132

Signed-off-by: Jakub Sokołowski <jakub@status.im>
 2023-09-06 10:18:44 +02:00
Jakub Sokołowski a823709dfe
add macm1-01.ih-eu-mda1.nimbus.prater host 
Replacement for `macos-01.ms-eu-dublin.nimbus.prater`.

https://github.com/status-im/infra-nimbus/issues/132

Signed-off-by: Jakub Sokołowski <jakub@status.im>
 2023-08-18 21:46:25 +02:00
Jakub Sokołowski 2a99b6ab43
nimbus.prater: add Nethermind node on linux-04 
https://github.com/status-im/infra-eth2/issues/11

Signed-off-by: Jakub Sokołowski <jakub@status.im>
 2023-08-17 13:26:08 +02:00
Jakub Sokołowski c662e92d51
nimbus.prater: configure Nethermind metrics endpoint 
https://github.com/status-im/infra-eth2/issues/11

Signed-off-by: Jakub Sokołowski <jakub@status.im>
 2023-08-17 13:05:28 +02:00
Jakub Sokołowski 8b6a22110a
nimbus.prater: open ports for Nethermind EL node 
https://github.com/status-im/infra-eth2/issues/11

Signed-off-by: Jakub Sokołowski <jakub@status.im>
 2023-08-11 14:21:19 +02:00
Jakub Sokołowski 1e24f891fe
nimbus.prater: add DNS discovery entry for nethermind 
https://github.com/status-im/infra-eth2/issues/11

Signed-off-by: Jakub Sokołowski <jakub@status.im>
 2023-08-11 14:20:34 +02:00
Jakub Sokołowski 600c6b02df
nimbus.prater: add chronos node on linux-06 host 
For Eugene for tracking regressions in chronos library.

Signed-off-by: Jakub Sokołowski <jakub@status.im>
 2023-08-08 15:54:11 +02:00
Jakub Sokołowski 8d5d8a3935
refactor handling of long libp2p branch name 
Signed-off-by: Jakub Sokołowski <jakub@status.im>
 2023-08-08 15:22:03 +02:00
Jakub Sokołowski 5344827479
nimbus.prater: deploy Nethermind node on linux-02 
Part of work to use Nethermind for eth2.prod fleet:
https://github.com/status-im/infra-eth2/issues/11

Signed-off-by: Jakub Sokołowski <jakub@status.im>
 2023-08-07 23:05:33 +02:00
Jakub Sokołowski 5586db729d
nimbus.prater: reduce max_headers_size to 128 KB 
Probably the reason for elevated memory usage when using Validator
Client with large number of validators attached.

Signed-off-by: Jakub Sokołowski <jakub@status.im>
 2023-08-07 14:54:07 +02:00
Jakub Sokołowski 2c9bbe832b
nimbus.sepolia: open Geth Websocket ports for Vac 
Requested by p1ge0nh8er for vacdev.misc host.

Signed-off-by: Jakub Sokołowski <jakub@status.im>
 2023-07-16 10:57:40 +02:00
Jakub Sokołowski f304db1cc0
nimbus.prater: bump Eirgon memory limit to 15% 
Signed-off-by: Jakub Sokołowski <jakub@status.im>
 2023-07-16 10:18:40 +02:00
Jakub Sokołowski 41025265e2
nimbus.prater: add 16 GB SWAP file, no SWAP partition 
There were OOM killer logs on `linux-06` due to Erigon.

Signed-off-by: Jakub Sokołowski <jakub@status.im>
 2023-07-12 09:38:19 +02:00
Jakub Sokołowski 7dff81fb2f
add windows-01.ih-eu-mda1.nimbus.prater host 
https://github.com/status-im/infra-nimbus/issues/132

Signed-off-by: Jakub Sokołowski <jakub@status.im>
 2023-07-07 12:17:33 +02:00
Jakub Sokołowski 4480d292be
nimbus.sepolia: debug flag for old attestation stability 
As requested by Dustin.

Signed-off-by: Jakub Sokołowski <jakub@status.im>
 2023-07-06 08:32:55 +02:00
Jakub Sokołowski cec778f4f1
nimbus.prater: debug flag for old attestation stability 
As requested by Dustin.

Signed-off-by: Jakub Sokołowski <jakub@status.im>
 2023-07-06 08:32:54 +02:00
Jakub Sokołowski 52e518d3c3
nimbus.sepolia: drop nim_commit=version-1-6 flag 
It no longer has any effect.

Signed-off-by: Jakub Sokołowski <jakub@status.im>
 2023-07-06 08:32:53 +02:00
Jakub Sokołowski 9455a6a5ae
nimbus.prater: drop nim_commit=version-1-6 flag 
It no longer has any effect.

Signed-off-by: Jakub Sokołowski <jakub@status.im>
 2023-07-06 08:29:09 +02:00
Jakub Sokołowski 44d77c54f4
nimbus.prater: move validators from Hetzner hosts 
The InnovaHosting servers are almost fully synced, time to move.
https://github.com/status-im/infra-nimbus/issues/132

Signed-off-by: Jakub Sokołowski <jakub@status.im>
 2023-07-06 08:29:08 +02:00
kdeme e0140a1497
nimbus.fluffy: drop bootstrap nodes, use network flag 
The `testnet0` network used with `--network` flag already includes our
own Fluffy nodes from this fleet, so no need to define them separately.

Signed-off-by: Jakub Sokołowski <jakub@status.im>
 2023-06-29 18:23:53 +02:00
Jakub Sokołowski f24fa170ee
add metal-01.ih-eu-mda1.nimbus.eth1 host 
https://github.com/status-im/infra-nimbus/issues/132

Signed-off-by: Jakub Sokołowski <jakub@status.im>
 2023-06-26 21:59:13 +02:00
Jakub Sokołowski ee7675543a
nimbus-geth-goerli: fix paths for extra data volume 
Otherwise `/docker` is part of root and there's not enough space.

Signed-off-by: Jakub Sokołowski <jakub@status.im>
 2023-06-26 12:54:54 +02:00
Jakub Sokołowski 5b85f5c26d
add three InnovaHosting servers to nimbus.geth fleet 
https://github.com/status-im/infra-nimbus/issues/132

Signed-off-by: Jakub Sokołowski <jakub@status.im>
 2023-06-23 21:15:19 +02:00
Jakub Sokołowski de28d46133
nimbus.mainnet: enable yamux for 2 libp2p nodes 
https://github.com/status-im/infra-nimbus/issues/148

Signed-off-by: Jakub Sokołowski <jakub@status.im>
 2023-06-23 15:14:42 +02:00
Jakub Sokołowski 052a8485aa
nimubs.prater: add 4 new hosts from InnovaHosting 
https://github.com/status-im/infra-nimbus/issues/132

Signed-off-by: Jakub Sokołowski <jakub@status.im>
 2023-06-22 20:44:31 +02:00
Jakub Sokołowski 766092f537
nimbus.prater: fix validator monitor details condition 
It was causing very high cardinality due to lots of validators.
Crashed our Hetzner metrics server.

Signed-off-by: Jakub Sokołowski <jakub@status.im>
 2023-06-22 12:07:16 +02:00
Jakub Sokołowski 7d9ebb1744
nimbus.prater: double beacon node open files limit 
Causes issues with metrics responses due to inability to open more files.

Signed-off-by: Jakub Sokołowski <jakub@status.im>
 2023-06-22 09:31:22 +02:00
Jakub Sokołowski 5f151fb828
all: grant admin rights to cheatfate user 
Signed-off-by: Jakub Sokołowski <jakub@status.im>
 2023-06-21 16:50:17 +02:00
Jakub Sokołowski 6f0ff4ad19
nimbus.prater: run only two nodes on MacOS host 
Not enough disk space for 3 nodes.

Signed-off-by: Jakub Sokołowski <jakub@status.im>
 2023-06-20 19:47:20 +02:00
Jakub Sokołowski 1aa507c359
prater: move validators to new linux-01 and linux-02 
https://github.com/status-im/infra-nimbus/issues/132

Signed-off-by: Jakub Sokołowski <jakub@status.im>
 2023-06-16 12:57:24 +02:00
Jakub Sokołowski 43b238c87e
nimbus.all: use new --validator-monitor-details flag 
Replaced `--validator-monitor-totals` in:
https://github.com/status-im/nimbus-eth2/pull/4468

Signed-off-by: Jakub Sokołowski <jakub@status.im>
 2023-06-07 19:04:37 +02:00
Etan Kissling b33a35c5e1
nimbus: set --rest-allow-origin='*' for public_api 
To allow browser based clients to access our API servers.

Depends on:

- https://github.com/status-im/infra-role-beacon-node-linux/pull/14
- https://github.com/status-im/infra-role-beacon-node-linux/pull/15
- https://github.com/status-im/infra-role-beacon-node-macos/pull/7
- https://github.com/status-im/infra-role-beacon-node-macos/pull/8
- https://github.com/status-im/infra-role-beacon-node-windows/pull/6
- https://github.com/status-im/infra-role-beacon-node-windows/pull/7

Signed-off-by: Etan Kissling <etan@status.im>
Signed-off-by: Jakub Sokołowski <jakub@status.im>
 2023-06-05 13:45:55 +02:00
Jakub Sokołowski e08586be04
nimbus.sepolia: move validators to new Innova host 
https://github.com/status-im/infra-nimbus/issues/132

Signed-off-by: Jakub Sokołowski <jakub@status.im>
 2023-06-05 12:08:01 +02:00
Jakub Sokołowski 6ae7dc00f0
nimbus.prater: add Innova host config 
https://github.com/status-im/infra-nimbus/issues/132

Signed-off-by: Jakub Sokołowski <jakub@status.im>
 2023-06-02 10:35:18 +02:00
Jakub Sokołowski da40f535ba
nimbus.sepolia: add Innova host config 
https://github.com/status-im/infra-nimbus/issues/132

Signed-off-by: Jakub Sokołowski <jakub@status.im>
 2023-06-02 10:34:20 +02:00
Jakub Sokołowski 948b5885fe
nimbus.prater: move disabling discovery to Hetzner group 
Signed-off-by: Jakub Sokołowski <jakub@status.im>
 2023-05-29 21:20:03 +02:00
Jakub Sokołowski e9a05d73ab
nimbus.prater: add DNS discovery for Erigon 
Necessary because we disable normal discovery.

Source:
https://github.com/ethereum/discv4-dns-lists

Signed-off-by: Jakub Sokołowski <jakub@status.im>
 2023-05-29 21:03:53 +02:00
Jakub Sokołowski a8741b7019
nimbus.prater: , disable discovery for Erigon 
It appears peer discovery can be detected by hetzner as an attempt at
network scanning:
https://github.com/ledgerwatch/erigon/issues/7234

Which can lead to the server being shut down.
For this reason I'm lowering max peers and disabling discovery.

Signed-off-by: Jakub Sokołowski <jakub@status.im>
 2023-05-29 20:06:58 +02:00
Jakub Sokołowski 3f3d763990
nimbus.prater: double nofile limit to 32768 
Which needs to be more than 4 x 7899 validators.

Signed-off-by: Jakub Sokołowski <jakub@status.im>
 2023-05-29 18:36:09 +02:00
Jakub Sokołowski e5324beaaa
nimbus-geth-goerli: remove one of three Geth nodes 
Not nough storage space to run 3 nodes on the hosts.

Also made AWS, Windows and MacOS nodes discover available
Geth nodes using Consul catalog.

Signed-off-by: Jakub Sokołowski <jakub@status.im>
 2023-05-24 18:34:03 +02:00
Jakub Sokołowski 597b85a0c8
nimbus.sepolia: open EL WebSocket to wakuv2.test 
https://github.com/waku-org/nwaku/issues/1624
https://github.com/status-im/infra-nim-waku/pull/69

Signed-off-by: Jakub Sokołowski <jakub@status.im>
 2023-05-17 18:15:47 +02:00
Jakub Sokołowski 2e14ecc79e
nimbus.sepolia: enable SMART metrics service 
Signed-off-by: Jakub Sokołowski <jakub@status.im>
 2023-05-17 18:11:27 +02:00
Jakub Sokołowski c15e1ea358
nimbus.mainnet: 10k max peers for stable-02@linux-03 
Signed-off-by: Jakub Sokołowski <jakub@status.im>
 2023-05-16 12:38:24 +02:00
Jakub Sokołowski a270f4044c
all: get Consul certificates from BitWarden 
Signed-off-by: Jakub Sokołowski <jakub@status.im>
 2023-05-16 12:14:38 +02:00
Jakub Sokołowski 0cd6de1ea6
nimbus.mainnet: use ncli_db from stable node 
Signed-off-by: Jakub Sokołowski <jakub@status.im>
 2023-05-16 12:14:27 +02:00
Jakub Sokołowski d04a98ea11
remove metal-01.he-eu-hel1.nimbus.mainnet 
https://github.com/status-im/infra-nimbus/issues/132

Signed-off-by: Jakub Sokołowski <jakub@status.im>
 2023-05-16 11:05:30 +02:00
Jakub Sokołowski a145963a2a
ih-eu-mda1: disable Consul check for SMART metrics 
The service fails on InnovaHosting servers due to HBA controller:
https://github.com/prometheus-community/smartctl_exporter/issues/26

Signed-off-by: Jakub Sokołowski <jakub@status.im>
 2023-05-15 19:51:41 +02:00
Jakub Sokołowski 0256a21dd0
nimbus.mainnet: add two more hosts from Innova 
The `linux-06` is a new host, while `linux-07` is re-purposed `linux-01`
from the prater fleet. This is done because the Innova company is out of
Xeon E5-2690 CPUs, and we want consistent performance across the fleet.

This way all seven hosts in `nimbus.mainnet` will use the E5-2690 CPU.

https://github.com/status-im/infra-nimbus/issues/132

Signed-off-by: Jakub Sokołowski <jakub@status.im>
 2023-05-09 15:07:22 +02:00
Jakub Sokołowski 406ac58f16
nimbus.mainnet: use --no-el for even number nodes 
Attempted fix issues with syncing where latency on the Geth endpoint is
too high for beacon nodes to finish syncing properly.

Signed-off-by: Jakub Sokołowski <jakub@status.im>
 2023-05-05 20:58:53 +02:00
Jakub Sokołowski 01cf22c9f8
get-geth-api-urls: rename variables for clarity 
Signed-off-by: Jakub Sokołowski <jakub@status.im>
 2023-05-05 19:21:49 +02:00
Jakub Sokołowski 5635b39e50
mainnet: decommission 6 Hetzner hosts 
Being replaced by InnovaHosting servers
https://github.com/status-im/infra-nimbus/issues/132

I'm leaving old `metal-07` as `metal-01` since we don't have a server
with `libp2p` nodes on mainnet yet.

Signed-off-by: Jakub Sokołowski <jakub@status.im>
 2023-04-28 12:22:46 +02:00
Jakub Sokołowski cd0841eba2
ih-eu-mda1: fix destintion DC for sending logs 
Signed-off-by: Jakub Sokołowski <jakub@status.im>
 2023-04-25 18:06:00 +02:00
Jakub Sokołowski e43ede93a5
nimbus.mainnet: fix exec layer URL logic 
Signed-off-by: Jakub Sokołowski <jakub@status.im>
 2023-04-20 22:28:26 +02:00
Jakub Sokołowski 940b2577da
nimbus.mainnet: add node layout for new Innova hosts 
Once they are synced and working we'll remove the Hetzner ones.

https://github.com/status-im/infra-nimbus/issues/132

Signed-off-by: Jakub Sokołowski <jakub@status.im>
 2023-04-18 18:14:53 +02:00
Jakub Sokołowski 8ddbd86139
nimbus.mainnet: revert removing of stable-02 node 
This reverts commit ab266d132d.

Signed-off-by: Jakub Sokołowski <jakub@status.im>
 2023-04-17 20:06:57 +02:00
Jakub Sokołowski 53ccd34927
nimbus.prater: bootstrap new Innova servers 
https://github.com/status-im/infra-nimbus/issues/132

Signed-off-by: Jakub Sokołowski <jakub@status.im>
 2023-04-17 17:11:27 +02:00
Jakub Sokołowski 1f5ad05893
nimbus.fluffy: 24h to make healthcheck critical 
This is not a critical service and there's a LOT of sevices that cause a
lot of alerts when developers are stopping and starting them.

Signed-off-by: Jakub Sokołowski <jakub@status.im>
 2023-04-12 19:28:37 +02:00
Jakub Sokołowski 9ec35f149b
nimbus.fluff: deploy smart-metrics exporter 
Signed-off-by: Jakub Sokołowski <jakub@status.im>
 2023-04-12 13:24:29 +02:00
Jakub Sokołowski ab266d132d
nimbus.mainnet: remove stable-02 node to save space 
We can always re-add it when we deploy to our new hosts.

Signed-off-by: Jakub Sokołowski <jakub@status.im>
 2023-04-07 12:00:20 +02:00
Jakub Sokołowski d41b3c89f8
dash.nimbus: increase logs retention to 30 days 
Signed-off-by: Jakub Sokołowski <jakub@status.im>
 2023-04-05 11:04:09 +02:00
Jakub Sokołowski d6de0faf1a
nimbus.prater: open Erigon metrics port on VPN 
Signed-off-by: Jakub Sokołowski <jakub@status.im>
 2023-03-23 15:14:38 +01:00
Jakub Sokołowski 0956c88d9a
nimbus.prater: enable metrics for Erigon nodes 
Signed-off-by: Jakub Sokołowski <jakub@status.im>
 2023-03-22 13:47:15 +01:00
Jakub Sokołowski 9aa83f8d47
nimbus.prater: use Erigon EL node if available 
Beginning of work on testing multi-EL node setup:
https://github.com/status-im/infra-nimbus/issues/144

Signed-off-by: Jakub Sokołowski <jakub@status.im>
 2023-03-22 12:10:34 +01:00
Jakub Sokołowski cdf362f2a9
nimbus.prater: provide private key for mining 
Signed-off-by: Jakub Sokołowski <jakub@status.im>
 2023-03-21 13:07:23 +01:00
Jakub Sokołowski f92bcb2802
nimbus.prater: fix erigon metrics port conflict 
Signed-off-by: Jakub Sokołowski <jakub@status.im>
 2023-03-21 13:07:13 +01:00
Jakub Sokołowski 4a5b7d7107
nimbus.prater: deploy two Erigon nodes 
Part of work to support multiple EL nodes driven by one CL node:
https://github.com/status-im/infra-nimbus/issues/144

Disabled Geth websocket as unused and conflicting with Erigon.

Signed-off-by: Jakub Sokołowski <jakub@status.im>
 2023-03-20 22:27:50 +01:00
Jakub Sokołowski 4ea58aeda7
deploy SMART metrics to all metal hosts 
Signed-off-by: Jakub Sokołowski <jakub@status.im>
 2023-03-18 18:02:58 +01:00
Jakub Sokołowski e1e21119da
use new exec layer variables for beacon nodes 
Role changes:
- https://github.com/status-im/infra-role-beacon-node-linux/commit/99a445d1
- https://github.com/status-im/infra-role-beacon-node-macos/commit/eecf9fea
- https://github.com/status-im/infra-role-beacon-node-windows/commit/dba21b36

Signed-off-by: Jakub Sokołowski <jakub@status.im>
 2023-03-16 00:59:48 +01:00
Jakub Sokołowski 5783656224
nimbus.mainnet: make AWS nodes have archive history 
Signed-off-by: Jakub Sokołowski <jakub@status.im>
 2023-03-02 12:22:51 +01:00
Jakub Sokołowski 768ecef371
nimbus.prater: use AWS hosts to sync first node 
Signed-off-by: Jakub Sokołowski <jakub@status.im>
 2023-03-02 12:22:45 +01:00
Jakub Sokołowski 090c4661a9
nimbus.mainnet: use AWS hosts to sync first node 
Signed-off-by: Jakub Sokołowski <jakub@status.im>
 2023-03-02 12:22:44 +01:00
Jakub Sokołowski 10f3132bcd
logs.nimbus: use G1 garbage collector 
https://github.com/status-im/infra-nimbus/issues/138

Signed-off-by: Jakub Sokołowski <jakub@status.im>
 2023-03-01 18:44:02 +01:00
Jakub Sokołowski 4e20fd2c14
nimbus.prater: use a new set of MEV boost URLs 
The one from `flashbots.net` seems to have connectivity issues:
```
REST request was interrupted
```

Signed-off-by: Jakub Sokołowski <jakub@status.im>
 2023-02-24 13:16:06 +01:00
Jakub Sokołowski 15d96ca3be
nimbus.mainnet/prater: use more threads than 1 
This setting is quite old and it's time to use more than one thread.
The `libp2p` branch is the exception that will keep using 1 thread.

Signed-off-by: Jakub Sokołowski <jakub@status.im>
 2023-02-24 11:37:17 +01:00
Jakub Sokołowski 3e3cff6840
nimbus.mainnet/prater: enable pruned history 
For most nodes this is fine, except for nodes with publicly exposed API,
which we should keep as synced from genesis and in archive mode.

Every other node is fine being prunned. I'm keeping AWS nodes not pruned
as fallback since increasing their storage is quick and easy.

Signed-off-by: Jakub Sokołowski <jakub@status.im>
 2023-02-21 22:52:41 +01:00
Jakub Sokołowski 6e05ef6eec
nimbus.prater: add linux-01.cs-eu-nld1.nimbus.prater 
Part of research into alternative hosting for Nimbus:
https://github.com/status-im/infra-nimbus/issues/132

Signed-off-by: Jakub Sokołowski <jakub@status.im>
 2023-02-21 21:25:42 +01:00
Jakub Sokołowski d6da1ce336
nimbus.mainnet: drop db_purge and db_sync vars 
Signed-off-by: Jakub Sokołowski <jakub@status.im>
 2023-02-21 12:38:17 +01:00
Jakub Sokołowski 297f347934
drop linux-01.cs-eu-nld1.nimbus.prater host 
Decided against using it due to maintenance costs:
https://github.com/status-im/infra-nimbus/issues/132

Signed-off-by: Jakub Sokołowski <jakub@status.im>
 2023-02-21 12:37:15 +01:00
Jakub Sokołowski 77278c2b65
logs.nimbus: lower ES JVM heap size to 40 GB 
Never ending issues with `Data too big` errors:
https://github.com/status-im/infra-nimbus/issues/138

Signed-off-by: Jakub Sokołowski <jakub@status.im>
 2023-02-21 00:11:55 +01:00
Jakub Sokołowski fc236d0132
remove SSH access from mamy, miran, s1fr0 
Signed-off-by: Jakub Sokołowski <jakub@status.im>
 2023-02-17 16:35:09 +01:00
Jakub Sokołowski c05fad0e01
add linux-01.cs-eu-nld1.nimbus.prater host 
Test for replacement of Hetzner hosts:
https://github.com/status-im/infra-nimbus/issues/132

Signed-off-by: Jakub Sokołowski <jakub@status.im>
 2023-01-30 21:34:21 +01:00
Jakub Sokołowski 61d586fa8c
logs.nimbus: set static max heap size to 48 GB 
Trying to fix the `Data too large` circuit breaker errors:
https://github.com/status-im/infra-nimbus/issues/138

Signed-off-by: Jakub Sokołowski <jakub@status.im>
 2023-01-28 14:17:04 +01:00
Jakub Sokołowski ee1ed5203b
logs/dash.nimbus: drop manual JVM heap size setting 
Lets try how well the automatic sizing works:
https://www.elastic.co/guide/en/elasticsearch/reference/7.17/important-settings.html#heap-size-settings

https://github.com/status-im/infra-nimbus/issues/138

Signed-off-by: Jakub Sokołowski <jakub@status.im>
 2023-01-19 15:00:14 +01:00
Jakub Sokołowski 9f0ab35e5d
requirements: bump elasticsearch and kiban to 7.17.8 
Signed-off-by: Jakub Sokołowski <jakub@status.im>
 2023-01-19 14:08:33 +01:00
Nikola K 2c3a01e1fe
add crypt1d as new nimbus devops 
Signed-off-by: Jakub Sokołowski <jakub@status.im>
 2023-01-13 10:46:17 +01:00
Jakub Sokołowski c18ead4714
nimbus.prater: add resync timer configuration 
Signed-off-by: Jakub Sokołowski <jakub@status.im>
 2023-01-04 17:24:02 +01:00
Jakub Sokołowski cd982c44f3
drop unused nimbus-db-purge role in favor of resync 
Signed-off-by: Jakub Sokołowski <jakub@status.im>
 2023-01-03 15:00:39 +01:00
Jakub Sokołowski d0bb5fa009
nimbus.ropsten: decomission, network shutdown 
https://blog.ethereum.org/2022/11/30/ropsten-shutdown-announcement

Signed-off-by: Jakub Sokołowski <jakub@status.im>
 2022-12-09 08:51:13 +01:00
Jakub Sokołowski 7f5463fe14
nimbus.prater: use common ERA files folder 
Signed-off-by: Jakub Sokołowski <jakub@status.im>
 2022-12-08 13:37:28 +01:00
Jakub Sokołowski 6cb88c3388
nimbus.ropsten: ropsten will be removed from unstable 
https://github.com/status-im/nimbus-eth2/pull/4280

Signed-off-by: Jakub Sokołowski <jakub@status.im>
 2022-12-01 08:46:47 +01:00
Zahary Karadjov 01b6f0b6c7
Enable payload builder on 06-prater-unstable 2022-11-30 13:44:27 +02:00
Zahary Karadjov 3f94bbfc0f
Configure the --suggested-fee-recipient option for BNs and VCs across the fleet 
Signed-off-by: Jakub Sokołowski <jakub@status.im>
 2022-11-30 10:49:23 +01:00
Jakub Sokołowski 86a5a83c99
nimbus.mainnet: use unstable ncli_db for ERA files 
Signed-off-by: Jakub Sokołowski <jakub@status.im>
 2022-11-30 10:46:37 +01:00
Zahary Karadjov 2fba850aca
Enable the VC payload builder on linux-06's testing intance 
Signed-off-by: Jakub Sokołowski <jakub@status.im>
 2022-11-24 14:16:22 +01:00
Jakub Sokołowski 8d25970d54
nimbus.prater: use unstable ncli_db to include fixes 
Specifically fixes for ERA file validating and naming.
https://github.com/status-im/nimbus-eth2/pull/4351

Signed-off-by: Jakub Sokołowski <jakub@status.im>
 2022-11-24 13:53:20 +01:00
Jakub Sokołowski fff4ae5c0a
nimbus.prater: enable nimbus-era-files role 
Necessary to generate ERA files for hosting.

Signed-off-by: Jakub Sokołowski <jakub@status.im>
 2022-11-08 17:56:03 +01:00
Jakub Sokołowski 7f7e234f36
nimbus.prater: bump header body and header limits 
Necessary to accomodate large number of validators handled by the
validator client node.

Signed-off-by: Jakub Sokołowski <jakub@status.im>
 2022-10-28 17:00:35 +02:00
Jakub Sokołowski 063a3b7c05
nimbus.prater: bump VC log level to DEBUG 
Signed-off-by: Jakub Sokołowski <jakub@status.im>
 2022-10-28 17:00:22 +02:00
Jakub Sokołowski 26f0db1807
nimbus.prater: enable VC for linux-06 libp2p node 
Signed-off-by: Jakub Sokołowski <jakub@status.im>
 2022-10-28 10:20:39 +02:00
Jakub Sokołowski 8f326fdf2a
logs.nimbus: open ES API port for Logstash hosts 
Signed-off-by: Jakub Sokołowski <jakub@status.im>
 2022-10-27 16:33:37 +02:00
Jakub Sokołowski 7e4f19b453
nimbus.prater: adjust validators count on last host 
Signed-off-by: Jakub Sokołowski <jakub@status.im>
 2022-10-26 11:15:50 +02:00
Jakub Sokołowski 2320bfd74a
nimbus.prater: open geth exporter ports 
Signed-off-by: Jakub Sokołowski <jakub@status.im>
 2022-10-25 21:11:47 +02:00
Jakub Sokołowski 4d43415e61
nimbus.ropsten: open geth exporter ports 
Signed-off-by: Jakub Sokołowski <jakub@status.im>
 2022-10-25 21:11:46 +02:00
Jakub Sokołowski c696c70cfc
nimbus.prater: fix beacon node validator distribution 
Missing part that already exists for Sepolia and Ropstne.

Signed-off-by: Jakub Sokołowski <jakub@status.im>
 2022-10-25 18:19:23 +02:00
Jakub Sokołowski baf158e399
nimbus.prater: drop index from validator client name 
Signed-off-by: Jakub Sokołowski <jakub@status.im>
 2022-10-24 18:23:55 +02:00
Jakub Sokołowski cf8bab14ab
nimbus.prater: use VC for unstable node on linux-03 
https://github.com/status-im/infra-nimbus/issues/122

Signed-off-by: Jakub Sokołowski <jakub@status.im>
 2022-10-24 16:09:42 +02:00
Jakub Sokołowski 439d07db86
nimbus.ropsten: reduce REST headers and body size 
This was a fix for very large URL arguments caused by large number of
validators applied to the connected validator client.
Unfortunately the values used were in bytes and not kilobytes.
In result we got unnecessarily high memory usage.

Signed-off-by: Jakub Sokołowski <jakub@status.im>
 2022-10-22 12:33:23 +02:00
Jakub Sokołowski 83073a7567
nimbus.mainnet: deploy libp2p nodes to metal-07 
Because `libp2p` nodes take up space on 05 and 06.

Signed-off-by: Jakub Sokołowski <jakub@status.im>
 2022-10-19 12:16:50 +02:00
Jakub Sokołowski c610692511
set explicit but wider values for websocket origins 
Possible fix for `nim-waku` and `geth` connectivity issues:
https://github.com/status-im/nwaku/issues/1269

Signed-off-by: Jakub Sokołowski <jakub@status.im>
 2022-10-17 19:17:03 +02:00
Jakub Sokołowski d63bd18938
give SSH access to p1ge0nh8er and s1fr0 
For debugging issues with Geth Websocket port for Nim-Waku.

Signed-off-by: Jakub Sokołowski <jakub@status.im>
 2022-10-14 15:54:55 +02:00
Jakub Sokołowski 4017674931
add systemd-journal groups to all users by default 
https://github.com/status-im/infra-role-bootstrap-linux/commit/d85963fa

Signed-off-by: Jakub Sokołowski <jakub@status.im>
 2022-10-14 15:01:31 +02:00
Jakub Sokołowski c8e3232dff
nimbus-era-files: use --network flag to fix errors 
Changes to default runtime config have caused errors like this:
```
FAT Could not load head state, database corrupt? tid=3639597 file=blockchain_dag.nim:920 head=e7d4f3b7:4636671 tail=4d611d5b:0 genesis=4d611d5b:0
```

Signed-off-by: Jakub Sokołowski <jakub@status.im>
 2022-10-12 09:30:46 +02:00
Jakub Sokołowski 5de206719e
nimbus.prater: temporary fix for buggy doppelganger 
https://github.com/status-im/infra-nimbus/issues/122

Signed-off-by: Jakub Sokołowski <jakub@status.im>
 2022-10-11 14:23:29 +02:00
Jakub Sokołowski 269a76a23e
nimbus.prater: deploy validator client to linux-04 
https://github.com/status-im/infra-nimbus/issues/122

Signed-off-by: Jakub Sokołowski <jakub@status.im>
 2022-10-11 13:39:59 +02:00
Jakub Sokołowski b5a6665de3
nimbus.ropsten: drop TTD config from extra flags 
Signed-off-by: Jakub Sokołowski <jakub@status.im>
 2022-10-05 15:18:29 +02:00
Jakub Sokołowski 1773ffb439
nimbus.prater: payload builder on unstable-large-01 
https://github.com/status-im/infra-nimbus/issues/128

Signed-off-by: Jakub Sokołowski <jakub@status.im>
 2022-09-29 15:03:34 +02:00
Jakub Sokołowski 759b3c8bf3
nimbus.prater: enable full light client data 
Signed-off-by: Jakub Sokołowski <jakub@status.im>
 2022-09-28 15:24:36 +02:00
Jakub Sokołowski 7da6edf144
nimbus.sepolia: add validator client service 
For now only for the first node.
https://github.com/status-im/infra-nimbus/issues/122

Signed-off-by: Jakub Sokołowski <jakub@status.im>
 2022-09-28 02:23:12 +02:00