Observability with Prometheus and Grafana

Alertmanager Receivers

Alertmanager supports a variety of receiver integrations, which are used to send alerts to external systems like email, PagerDuty, OpsGenie, and many other notification systems.

To configure a receiver, you need to add a new section to the Alertmanager configuration file. This is an example of a configuration for an email receiver:

route:
  group_by: ['alertname']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 20m
  # define a receiver
  receiver: 'email-me'

receivers:
  - name: 'email-me'
    email_configs:
    - to: '$SMTP_TO'
      from: '$SMTP_FROM'
      smarthost: '$SMTP_SERVER:$SMTP_PORT'
      auth_username: '$SMTP_USER'
      auth_identity: '$SMTP_USER'
      auth_password: '$SMTP_PASS'

Here are the key Alertmanager routing parameters and what they control:

group_by:

A list of labels used to group alerts together. Alerts sharing the same values for all labels in group_by are treated as one group. This allows related alerts (for example, from the same service or job) to be batched into a single notification.

group_wait:

The amount of time Alertmanager waits before sending the first notification for a new alert group. This delay lets it collect additional alerts that arrive shortly after the first one, so they can be included in the same notification. For instance, if group_wait is set to 30s, Alertmanager waits 30 seconds before sending the initial notification for a group.

group_interval:

The minimum time between notifications for the same alert group. Once a notification for a group has been sent, Alertmanager waits at least this long before sending another notification for new alerts added to that group. For example, with group_interval: 5m, any new alerts that appear in an existing group will trigger another notification no sooner than 5 minutes later.

repeat_interval:

Specifies how often Alertmanager re-sends notifications for alerts that remain active. This serves as a reminder for ongoing issues. If repeat_interval is set to 1h, the same alert is re-notified every hour until it resolves.

Difference between group_interval and repeat_interval:

Both group_interval and repeat_interval control the timing of notifications; they can be confusing because they seem similar but serve different purposes:

• group_interval controls how frequently new alerts added to an existing group can trigger a new notification. It’s about updates to the group.

• repeat_interval controls how often notifications for already-firing alerts in that group are re-sent as reminders. It’s about ongoing, unchanged alerts.

Other configuration parameters are:

receiver:

The name of the receiver used to send notifications for the alerts.

receivers:

A list of receiver configurations that define how alerts should be sent to external systems. Each receiver configuration includes the name of the receiver, its type, and the necessary parameters for its integration.

email_configs, to, from, smarthost, auth_username, auth_identity, auth_password:

These are the necessary parameters for the email receiver integration, including the recipient email address, the sender email address, the SMTP server address, the SMTP username, and the SMTP password.

Let's see this in practice. Start by exporting some environment variables that contain the necessary parameters for the email receiver integration.

export SMTP_SERVER=""
export SMTP_PORT=""
export SMTP_USER=""
export SMTP_PASS=""
export SMTP_FROM=""
export SMTP_TO=""

Then, add the following configuration to the Alertmanager configuration file:

cat << EOF > /etc/alertmanager/alertmanager.yml
route:
  group_by: ['alertname']
  group_wait