diff --git a/ADVANCED.md b/ADVANCED.md index 51cc94a..1d68218 100644 --- a/ADVANCED.md +++ b/ADVANCED.md @@ -14,7 +14,8 @@ There are multiple environment variables you can configure to modify behaviour o * `RLN_RELAY_CRED_PASSWORD` - password for encrypting RLN credentials * `EXTRA_ARGS` - this variable allows you to specify additional or overriding CLI option for the Waku node which will be appended to the `wakunode2` command. (e.g. `EXTRA_ARGS="--store=false --max-connections=3000`) * `CERTS_DIR` - allows you to define a path where SSL certificates are/will be stored. It needs to follow the directory structure produced by Certbot in `/etc/letsencrypt` -* `STORATE_SIZE` - overrides the default allowed DB size of waku message storage service. Current default is 1GB. (e.g. `STORAGE_SIZE=2GB` or `STORAGE_SIZE=3500MB`) +* `STORAGE_SIZE` - overrides the default allowed DB size of Waku message storage service. Current default is 1GB. (e.g. `STORAGE_SIZE=2GB` or `STORAGE_SIZE=3500MB`) +* `PROMETHEUS_RETENTION_SIZE` - overrides the default allow data dir for Prometheus data (which feeds Grafana). Current default is 5GB. (e.g. `PROMETHEUS_RETENTION_SIZE=1GB`) ## Log monitoring and troubleshooting diff --git a/README.md b/README.md index 6e8a76d..6843ce4 100644 --- a/README.md +++ b/README.md @@ -193,6 +193,34 @@ sudo docker container prune sudo docker volume prune ``` +#### journal + +If your `/var/log` gets quite large: + +``` +journalctl --disk-usage +> Archived and active journals take up 1.5G in the file system. +``` + +You can cap the size in ` /etc/systemd/journald.conf` with + +``` +SystemMaxUse=50M +``` + +then restart to apply + +``` +systemctl restart systemd-journald +``` + +and verify +``` +journalctl --disk-usage +> Archived and active journals take up 55.8M in the file system. +``` + ----- + # FAQ [see](FAQ.md) diff --git a/docker-compose.yml b/docker-compose.yml index e839406..c7e0b73 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -81,6 +81,7 @@ services: - ./monitoring/prometheus-config.yml:/etc/prometheus/prometheus.yml:Z command: - --config.file=/etc/prometheus/prometheus.yml + - --storage.tsdb.retention.size=${PROMETHEUS_RETENTION_SIZE:-5GB} # ports: # - 127.0.0.1:9090:9090 restart: on-failure:5 diff --git a/set_storage_retention.sh b/set_storage_retention.sh index 1b81f2e..164b0ce 100755 --- a/set_storage_retention.sh +++ b/set_storage_retention.sh @@ -74,8 +74,8 @@ fi AVAIL_SPACE_MB=$(df -m . | tail -1 | awk '{ print $4 }' | sed 's/^\([0-9]\+\)M$/\1/') # Select a store size with the following constraints: -# - Margin: 1GB - 1GB will be left over for the system and logs -# - Min: 1GB - The minimum allocated space will be 1GB (old default) +# - Margin: 6GB - 1GB will be left over for the system, logs and prometheus data +# - Min: 1GB - The minimum allocated space will be 1GB # - Max: 30GB - The maximum allocated space will be 30GB select_store_size $AVAIL_SPACE_MB $PGSQL_SIZE_MB 1024 1024 $(( 30 * 1024 ))