History

Jakub Sokołowski fb7bde2377 dash.nimbus: lower indices kept to 10 We have bigger log volume now that we fixed the logging setup in: https://github.com/status-im/infra-nimbus/issues/81 Signed-off-by: Jakub Sokołowski <jakub@status.im>		2022-02-07 15:07:55 +01:00
..
defaults	logclean-job: increase timer timeout to 5 minutes	2021-02-27 15:26:53 +01:00
tasks	dash.nimbus: lower indices kept to 10	2022-02-07 15:07:55 +01:00
templates	logclean-job: fix bug in script that deletes newer first	2021-02-27 15:37:49 +01:00
README.md	logclean-job: keep only 9 newest logstash daily indices	2020-02-07 14:15:31 +01:00

README.md

Description

This is a systemd timer that runs daily and removes from ElasticSearch logs older than N days.

This is done because Nimbus generates a metric shitton of TRACE logs.

Configuration

The main configuration values are:

logclean_es_host: '127.0.0.1'
logclean_es_port: 9200
logclean_index_regex: 'logstash-*'
logclean_keep_indices: 120
logclean_service_name: 'logclean-job'
logclean_service_timeout: 60
logclean_timer_frequency: 'daily'

For sake of security minimum for logclean_keep_indices is 60.

Usage

To check the timer status use:

 $ sudo systemctl list-timers logclean-job.timer
NEXT                         LEFT     LAST PASSED UNIT               ACTIVATES
Sat 2020-02-08 00:00:00 UTC  10h left n/a  n/a    logclean-job.timer logclean-job.service

You can check job logs using:

 $ sudo journalctl -o cat -a -u logclean-job.service
...
Starting "Job for cleaning ElasticSearch cluster periodically."...
Checking ElasticSearch for indices to clean....
Nothing to remove. (3/90 indices)
Started "Job for cleaning ElasticSearch cluster periodically.".