Set retention for Prometheus data dir (#146)

ADVANCED.md

@@ -14,7 +14,8 @@ There are multiple environment variables you can configure to modify behaviour o
 * `RLN_RELAY_CRED_PASSWORD` - password for encrypting RLN credentials
 * `EXTRA_ARGS` - this variable allows you to specify additional or overriding CLI option for the Waku node which will be appended to the `wakunode2` command. (e.g. `EXTRA_ARGS="--store=false --max-connections=3000`)
 * `CERTS_DIR` - allows you to define a path where SSL certificates are/will be stored. It needs to follow the directory structure produced by Certbot in `/etc/letsencrypt`
-* `STORATE_SIZE` - overrides the default allowed DB size of waku message storage service. Current default is 1GB. (e.g. `STORAGE_SIZE=2GB` or `STORAGE_SIZE=3500MB`)
+* `STORAGE_SIZE` - overrides the default allowed DB size of Waku message storage service. Current default is 1GB. (e.g. `STORAGE_SIZE=2GB` or `STORAGE_SIZE=3500MB`)
+* `PROMETHEUS_RETENTION_SIZE` - overrides the default allow data dir for Prometheus data (which feeds Grafana). Current default is 5GB. (e.g. `PROMETHEUS_RETENTION_SIZE=1GB`)
 
 ## Log monitoring and troubleshooting
 

README.md

@@ -193,6 +193,34 @@ sudo docker container prune
 sudo docker volume prune
 ```
 
+#### journal
+
+If your `/var/log` gets quite large:
+
+```
+journalctl --disk-usage
+> Archived and active journals take up 1.5G in the file system.
+```
+
+You can cap the size in ` /etc/systemd/journald.conf` with
+
+```
+SystemMaxUse=50M
+```
+
+then restart to apply
+
+```
+systemctl restart systemd-journald
+```
+
+and verify
+```
+journalctl --disk-usage
+> Archived and active journals take up 55.8M in the file system.
+```
+
 -----
+
 # FAQ
 [see](FAQ.md)

docker-compose.yml

@@ -81,6 +81,7 @@ services:
       - ./monitoring/prometheus-config.yml:/etc/prometheus/prometheus.yml:Z
     command:
       - --config.file=/etc/prometheus/prometheus.yml
+      - --storage.tsdb.retention.size=${PROMETHEUS_RETENTION_SIZE:-5GB}
     # ports:
     #   - 127.0.0.1:9090:9090
     restart: on-failure:5

set_storage_retention.sh

@@ -74,8 +74,8 @@ fi
 AVAIL_SPACE_MB=$(df -m . | tail -1 | awk '{ print $4 }' | sed 's/^\([0-9]\+\)M$/\1/')
 
 # Select a store size with the following constraints:
-# - Margin: 1GB - 1GB will be left over for the system and logs
-# - Min: 1GB - The minimum allocated space will be 1GB (old default)
+# - Margin: 6GB - 1GB will be left over for the system, logs and prometheus data
+# - Min: 1GB - The minimum allocated space will be 1GB
 # - Max: 30GB - The maximum allocated space will be 30GB
 select_store_size $AVAIL_SPACE_MB $PGSQL_SIZE_MB 1024 1024 $(( 30 * 1024 ))