status-im
/
infra-nimbus
mirror of https://github.com/status-im/infra-nimbus.git
2
0
Fork 0
Code Issues Projects Releases Wiki Activity
1,175 Commits 10 Branches 0 Tags 31 MiB
Commit Graph

27 Commits

Author SHA1 Message Date
Siddarth Kumar 5c7a3314f1 vault: migrate secrets from bitwarden 2024-12-04 12:23:29 +05:30
Jakub Sokołowski 89c042cf06
logs.nimbus: fix hq.logs ipset name 
Signed-off-by: Jakub Sokołowski <jakub@status.im>
 2024-10-31 17:38:31 +01:00
Jakub Sokołowski e755ec67ae
all: replace old naming of hq.metrics fleet 
https://github.com/status-im/infra-hq/issues/158

Signed-off-by: Jakub Sokołowski <jakub@status.im>
 2024-10-18 10:18:39 +02:00
markoburcul e0b5b78720 logs: Migrate to nftables 
Referenced issue: https://github.com/status-im/infra-misc/issues/301

Signed-off-by: markoburcul <marko@status.im>
 2024-10-08 09:41:17 +02:00
Jakub Sokołowski dd218622e3
logs.nimbus: specify json.msg as default query field 
This avoids errors like this:
```
failed to create query: field expansion matches too many fields, limit: 1024, got: 1046
```
Which is caused by default being a wildcard, that causes querying of
available fields, which in case of Nimbus logs is more than 1024.

This could be fixed by increasing `max_clasue_count`, but a cleaner
solution is simply limiting queries without a field specified to `json.msg`.

https://www.elastic.co/guide/en/elasticsearch/reference/7.17/index-modules.html#dynamic-index-settings
https://www.elastic.co/guide/en/kibana/7.17/upgrade-assistant-api-default-field.html

Signed-off-by: Jakub Sokołowski <jakub@status.im>
 2024-09-25 14:33:00 +02:00
markoburcul c38dc75f7f
ansible: Change hostname to inventory_hostname 
Due to the change caused by updated terraform script, we need to change
usage of variable `hostname` to `inventory_hostname`.

Signed-off-by: markoburcul <marko@status.im>
 2024-09-16 08:10:36 +02:00
Jakub Sokołowski d2270feece
logs.nimbus: increase total_fields.limit to 1500 
This is a special case since these logs are all custom JSON, so
increasing this is fine for now. I can't control what they put in logs.
https://discuss.elastic.co/t/approaches-to-deal-with-limit-of-total-fields-1000-in-index-has-been-exceeded/241039

Signed-off-by: Jakub Sokołowski <jakub@status.im>
 2023-10-12 18:56:21 +02:00
Jakub Sokołowski 4ea58aeda7
deploy SMART metrics to all metal hosts 
Signed-off-by: Jakub Sokołowski <jakub@status.im>
 2023-03-18 18:02:58 +01:00
Jakub Sokołowski 10f3132bcd
logs.nimbus: use G1 garbage collector 
https://github.com/status-im/infra-nimbus/issues/138

Signed-off-by: Jakub Sokołowski <jakub@status.im>
 2023-03-01 18:44:02 +01:00
Jakub Sokołowski 77278c2b65
logs.nimbus: lower ES JVM heap size to 40 GB 
Never ending issues with `Data too big` errors:
https://github.com/status-im/infra-nimbus/issues/138

Signed-off-by: Jakub Sokołowski <jakub@status.im>
 2023-02-21 00:11:55 +01:00
Jakub Sokołowski 61d586fa8c
logs.nimbus: set static max heap size to 48 GB 
Trying to fix the `Data too large` circuit breaker errors:
https://github.com/status-im/infra-nimbus/issues/138

Signed-off-by: Jakub Sokołowski <jakub@status.im>
 2023-01-28 14:17:04 +01:00
Jakub Sokołowski ee1ed5203b
logs/dash.nimbus: drop manual JVM heap size setting 
Lets try how well the automatic sizing works:
https://www.elastic.co/guide/en/elasticsearch/reference/7.17/important-settings.html#heap-size-settings

https://github.com/status-im/infra-nimbus/issues/138

Signed-off-by: Jakub Sokołowski <jakub@status.im>
 2023-01-19 15:00:14 +01:00
Jakub Sokołowski 9f0ab35e5d
requirements: bump elasticsearch and kiban to 7.17.8 
Signed-off-by: Jakub Sokołowski <jakub@status.im>
 2023-01-19 14:08:33 +01:00
Jakub Sokołowski 8f326fdf2a
logs.nimbus: open ES API port for Logstash hosts 
Signed-off-by: Jakub Sokołowski <jakub@status.im>
 2022-10-27 16:33:37 +02:00
Jakub Sokołowski a6bc25ebd3
logs.nimbus: drop ElasticHQ in favor of ES Comrade 
https://github.com/status-im/infra-hq/issues/82

Signed-off-by: Jakub Sokołowski <jakub@status.im>
 2022-07-14 20:31:24 +02:00
Jakub Sokołowski 472a379a9a
dash.nimbus: fix LB variable typos 
Signed-off-by: Jakub Sokołowski <jakub@status.im>
 2022-07-14 16:51:20 +02:00
Jakub Sokołowski 07d561b62a
logs.nimbus: fix firewall rules for logs.hq fleet 
https://github.com/status-im/infra-hq/issues/80

Signed-off-by: Jakub Sokołowski <jakub@status.im>
 2022-07-11 17:42:50 +02:00
Miran 2b079f1774
fix various typos in comments and roles 2022-07-04 15:07:55 +02:00
Jakub Sokołowski c474a3311d
logs.nimbus: leave memory for Lucene, use 32 GB heap 
https://github.com/status-im/infra-nimbus/issues/96

"The moral of the story is this: even when you have memory to spare, try
to avoid crossing the 32 GB heap boundary. It wastes memory, reduces CPU
performance, and makes the GC struggle with large heaps."

https://www.elastic.co/guide/en/elasticsearch/guide/current/heap-sizing.html#compressed_oops
https://www.elastic.co/guide/en/elasticsearch/guide/current/heap-sizing.html#_give_less_than_half_your_memory_to_lucene

Signed-off-by: Jakub Sokołowski <jakub@status.im>
 2022-06-01 12:28:48 +02:00
Jakub Sokołowski 5e1af52f2d
logs.nimubs: move nimbus-es to our own SSL proxy 
Signed-off-by: Jakub Sokołowski <jakub@status.im>
 2022-06-01 11:43:45 +02:00
Jakub Sokołowski f8b09ccd3c
dash.nimbus: upgrade kibana and ES to 7.17.4 
Signed-off-by: Jakub Sokołowski <jakub@status.im>
 2022-06-01 11:40:19 +02:00
Jakub Sokołowski d231accd45
logs.nimbus: refactor OAuth container setup 
This is to take into account changes made in:
https://github.com/status-im/infra-role-oauth-proxy/commit/4dd42c56

Signed-off-by: Jakub Sokołowski <jakub@status.im>
 2022-04-11 11:26:38 +02:00
Jakub Sokołowski 89093c894a
logs.nimbus: firewall for ElasticSearch exporter port 
Signed-off-by: Jakub Sokołowski <jakub@status.im>
 2022-03-03 18:23:13 +01:00
Jakub Sokołowski 18eb46836e
logs.nimbus: use docker compose setup for elasticsearch 
https://github.com/status-im/infra-role-elasticsearch/commit/567f32ac

Signed-off-by: Jakub Sokołowski <jakub@status.im>
 2022-02-11 02:08:47 +01:00
Jakub Sokołowski 0398c31ad9
logs.nimbus: drop ES replicas down to 0 to save space 
https://github.com/status-im/infra-nimbus/issues/81

Signed-off-by: Jakub Sokołowski <jakub@status.im>
 2022-02-09 18:43:23 +01:00
Jakub Sokołowski ba2caedb64
logs.nimbus: bump JVM memory limits to 80% 
We started getting errors like:
```
Caused by: org.elasticsearch.common.breaker.CircuitBreakingException: [parent] Data too large, data for [internal:cluster/nodes/indices/shard/store[n]] would be [36997638608/34.4gb], which is larger than the limit of [36795685273/34.2gb], real usage: [36997638456/34.4gb], new bytes reserved: [152/152b], usages [request=0/0b, fielddata=0/0b, in_flight_requests=152/152b, model_inference=0/0b, eql_sequence=0/0b, accounting=3913236/3.7mb]
```

Signed-off-by: Jakub Sokołowski <jakub@status.im>
 2022-02-07 11:49:44 +01:00
Jakub Sokołowski ade0a50a53
logs.nimbus: migrate to Hetzner metal hosts 
For better performance, lower prices, and better network utilization.

https://github.com/status-im/infra-nimbus/issues/84

Signed-off-by: Jakub Sokołowski <jakub@status.im>
 2022-02-03 21:48:24 +01:00