status-im
/
infra-role-alertmanager
mirror of https://github.com/status-im/infra-role-alertmanager.git
2
0
Fork 0
Code Issues Projects Releases Wiki Activity
Ansible role for configuring AlertManager for Prometheus
36 Commits 1 Branch 0 Tags 104 KiB
Jinja 100%
Go to file
Jakub Sokołowski 19d63d6daa
docker: migrate to Docker Compose V2 
Signed-off-by: Jakub Sokołowski <jakub@status.im>
 		2024-10-15 21:37:22 +02:00
defaults docker: migrate to Docker Compose V2 2024-10-15 21:37:22 +02:00
files add general custom receivers and template 2024-03-28 11:01:40 +01:00
handlers add role files from infra-hq 2020-12-07 12:38:42 +01:00
meta meta: use full names of Ansible roles 2024-05-14 10:25:02 +02:00
tasks docker: migrate to Docker Compose V2 2024-10-15 21:37:22 +02:00
templates ansible: Change hostname to inventory_hostname 2024-09-17 08:05:06 +02:00
README.md replace Tinc VPN with WireGuard 2021-05-19 23:00:33 +02:00

README.md

Description

This role configures AlertManager to notify people of threshold breaches in rules configured in Prometheus master instance.

Configuration

The bare minimum should be:

alertmanager_domain: 'alerts.example.org'
alertmanager_admin_email: 'admin@example.org'

# SMTP fallback
alertmanager_admin_email: 'admin@example.org'
alertmanager_smtp_host: smtp.mail.example.org'
alertmanager_smtp_from: 'alerts@example.org'
alertmanager_smtp_user: 'secret-smtp-user'
alertmanager_smtp_pass: 'secret-smtp-pass'

# VictorOps API
alertmanager_victorops_api_key: 'secret-victorops-api-key'
alertmanager_victorops_routing_key:  'alert-manager'
alertmanager_victorops_service_url: 'https://alert.victorops.com/integrations/generic/123123123/alert/'

Take note you will have to create an alert-manager routing rule in VictorOps.

There is also optional OAuth Proxy configuration:

alertmanager_oauth_id: '123qwe123qwe'
alertmanager_oauth_secret: '123qwe123qwe123qwe123qwe'
alertmanager_oauth_cookie_secret: '123qwe'
alertmanager_oauth_gh_org: 'my-gh-org'

Management

You can manage existing alerts by using the amtool on any of the hosts running this:

 > amtool alert
Alertname        Starts At                Summary
Test_Alert       2018-07-06 18:30:18 UTC  This is a testing alert!

 > amtool silence
ID                                    Matchers                Starts At                Ends At                  Updated At               Created By  Comment  
9635b573-5177-4601-a3b0-ac6a25d0a4ef  alertname=InstanceDown  2018-07-06 12:37:04 UTC  2018-07-06 14:36:05 UTC  2018-07-06 12:37:04 UTC  jakubgs     test

Details

AlertManager runs in a cluster to achieve high availability. The peer connect via WireGuard VPN. The service listens on :9093 and the Prometheus instance connects to that port via the VPN to inform it of threshold breaches.

The main configuration resides in templates/alertmanager.yml.j2. It configures all the receivers of alerts generated by Prometheus master instance.

The are three main sections:

  • global - Configure general auth related options for SMTP and Slack receivers.
  • receivers - Defines destinations of alets which can be used in the route section.
  • route - Defines rules based on which alerts are directed to defined receivers.

For more details see: https://prometheus.io/docs/alerting/configuration/