nimbus-db-purge: add role to periodicall purge DB

https://github.com/status-im/infra-nimbus/issues/86

Signed-off-by: Jakub Sokołowski <jakub@status.im>

ansible/roles/nimbus-db-purge/README.md

@@ -0,0 +1,38 @@
+# Description
+
+This role configures a Systemd timer that periodically purges the Nimbus node database to test the sync process. Optionally it can use the [trusted node sync](https://nimbus.guide/trusted-node-sync.html) functionality.
+
+# Configuration
+
+In order to purge database of `beacon-node-mainnet-stable` weekly and sync it from trusted node:
+```yaml
+nimbus_db_purge_node_service_name: 'beacon-node-mainnet-stable'
+nimbus_db_purge_node_service_path: '/data/{{ nimbus_db_purge_node_service_name }}'
+nimbus_db_purge_node_network_name: 'mainnet'
+nimbus_db_purge_timer_frequency: 'weekly'
+nimbus_db_purge_truster_node_api_url: 'http://localhost:9300'
+nimbus_db_purge_trusted_node_sync_enabled: true
+```
+
+# Management
+
+The Systemd timer can be managed with `systemctl`:
+```
+ > sudo systemctl -o cat status purge-beacon-node-mainnet-stable
+● purge-beacon-node-mainnet-stable.service - Purge beacon-node-mainnet-stable DB
+     Loaded: loaded (/etc/systemd/system/purge-beacon-node-mainnet-stable.service; enabled; vendor preset: enabled)
+     Active: inactive (dead) since Mon 2022-02-28 16:26:19 UTC; 4min 35s ago
+TriggeredBy: ● purge-beacon-node-mainnet-stable.timer
+       Docs: https://github.com/status-im/infra-role-systemd-timer
+    Process: 1496098 ExecStart=/data/beacon-node-mainnet-stable/purge.sh (code=exited, status=0/SUCCESS)
+   Main PID: 1496098 (code=exited, status=0/SUCCESS)
+
+NOT 2022-02-28 16:26:19.669+00:00 Database initialized, historical blocks will be backfilled when starting the node missingSlots=3270047
+NOT 2022-02-28 16:26:19.669+00:00 Done, your beacon node is ready to serve you! Don't forget to check that you're on the canoncial chain by comparing the checkpoint root with other online sources. See https://nim>
+ >>> Starting node: beacon-node-mainnet-stable
+  nimbus : TTY=unknown ; PWD=/data/beacon-node-mainnet-stable ; USER=root ; COMMAND=/usr/bin/systemctl start beacon-node-mainnet-stable
+pam_unix(sudo:session): session opened for user root by (uid=0)
+pam_unix(sudo:session): session closed for user root
+purge-beacon-node-mainnet-stable.service: Succeeded.
+Finished Purge beacon-node-mainnet-stable DB.
+```

ansible/roles/nimbus-db-purge/defaults/main.yml

@@ -0,0 +1,15 @@
+---
+#nimbus_db_purge_node_service_name: ~
+#nimbus_db_purge_node_service_path: ~
+#nimbus_db_purge_node_network_name: ~
+nimbus_db_purge_trusted_node_sync_enabled: false
+#nimbus_db_purge_trusted_node_api_url: 'http://localhost:1234'
+
+nimbus_db_purge_timer_name: 'purge-{{ nimbus_db_purge_node_service_name }}'
+nimbus_db_purge_timer_description: 'Purge {{ nimbus_db_purge_node_service_name }} DB'
+nimbus_db_purge_timer_user: 'nimbus'
+nimbus_db_purge_timer_group: 'staff'
+nimbus_db_purge_timer_enabled: true
+nimbus_db_purge_timer_timeout: 600
+nimbus_db_purge_timer_frequency: 'weekly'
+nimbus_db_purge_timer_script_path: '{{ nimbus_db_purge_node_service_path | mandatory }}/purge.sh'

ansible/roles/nimbus-db-purge/tasks/main.yml

@@ -0,0 +1,24 @@
+---
+- name: Create purge and sync script
+  template:
+    src: 'purge_sync.sh.j2'
+    dest: '{{ nimbus_db_purge_timer_script_path }}'
+    owner: 'root'
+    group: 'staff'
+    mode: 0750
+
+- name: Create timer for purge script
+  include_role: name=systemd-timer
+  vars:
+    systemd_timer_name: '{{ nimbus_db_purge_timer_name | mandatory }}'
+    systemd_timer_description: '{{ nimbus_db_purge_timer_description | mandatory }}'
+    systemd_timer_user: '{{ nimbus_db_purge_timer_user }}'
+    systemd_timer_group: '{{ nimbus_db_purge_timer_group }}'
+    systemd_timer_start_on_creation: false
+    systemd_timer_ionice_class: 'idle'
+    systemd_timer_consul_warning: true
+    systemd_timer_enabled: '{{ nimbus_db_purge_timer_enabled }}'
+    systemd_timer_frequency: '{{ nimbus_db_purge_timer_frequency }}'
+    systemd_timer_timeout_sec: '{{ nimbus_db_purge_timer_timeout }}'
+    systemd_timer_work_dir: '{{ nimbus_db_purge_node_service_path }}'
+    systemd_timer_script_path: '{{ nimbus_db_purge_timer_script_path }}'

ansible/roles/nimbus-db-purge/templates/purge.sh.j2

@@ -0,0 +1,27 @@
+#!/usr/bin/env bash
+# vim:ft=sh
+set -Eeuo pipefail
+
+TRUSTED_NODE_API_URL='{{ nimbus_db_purge_trusted_node_api_url | mandatory }}'
+SERVICE_NAME='{{ nimbus_db_purge_node_service_name | mandatory }}'
+SERVICE_PATH='{{ nimbus_db_purge_node_service_path | mandatory }}'
+NETWORK_NAME='{{ nimbus_db_purge_node_network_name | mandatory }}'
+SERVICE_DATA_PATH="${SERVICE_PATH}/data/shared_${NETWORK_NAME}_0/"
+
+echo "Stopping node: ${SERVICE_NAME}"
+sudo systemctl stop "${SERVICE_NAME}"
+
+echo "Purging DB: ${SERVICE_DATA_PATH}"
+sudo rm -fr "${SERVICE_DATA_PATH}/db"
+
+{% if 
+echo "Syncing from trusted node: ${TRUSTED_NODE_API_URL}"
+sudo -u nimbus "${SERVICE_PATH}/repo/build/nimbus_beacon_node" \
+    trustedNodeSync \
+    --data-dir="${SERVICE_DATA_PATH}" \
+    --trusted-node-url="${TRUSTED_NODE_API_URL}" \
+    --network="${NETWORK_NAME}" \
+    --backfill=false
+
+echo "Starting node: ${SERVICE_NAME}"
+sudo systemctl start "${SERVICE_NAME}"

ansible/roles/nimbus-db-purge/templates/purge_sync.sh.j2

@@ -0,0 +1,28 @@
+#!/usr/bin/env bash
+# vim:ft=sh
+set -Eeuo pipefail
+
+SERVICE_NAME='{{ nimbus_db_purge_node_service_name | mandatory }}'
+SERVICE_PATH='{{ nimbus_db_purge_node_service_path | mandatory }}'
+NETWORK_NAME='{{ nimbus_db_purge_node_network_name | mandatory }}'
+SERVICE_DATA_PATH="${SERVICE_PATH}/data/shared_${NETWORK_NAME}_0/"
+
+echo " >>> Stopping node: ${SERVICE_NAME}"
+sudo systemctl stop "${SERVICE_NAME}"
+
+echo " >>> Purging DB: ${SERVICE_DATA_PATH}"
+rm -fr "${SERVICE_DATA_PATH}/db"
+
+{% if nimbus_db_purge_trusted_node_sync_enabled %}
+TRUSTED_NODE_API_URL='{{ nimbus_db_purge_truster_node_api_url | mandatory }}'
+echo " >>> Syncing from trusted node: ${TRUSTED_NODE_API_URL}"
+"${SERVICE_PATH}/repo/build/nimbus_beacon_node" \
+    trustedNodeSync \
+    --data-dir="${SERVICE_DATA_PATH}" \
+    --trusted-node-url="${TRUSTED_NODE_API_URL}" \
+    --network="${NETWORK_NAME}" \
+    --backfill=false
+
+{% endif %}
+echo " >>> Starting node: ${SERVICE_NAME}"
+sudo systemctl start "${SERVICE_NAME}"