Alertmanager receives the alerts that Prometheus fires, groups and deduplicates them, and delivers notifications to channels such as email, Slack or any webhook. Prometheus decides when something is wrong; Alertmanager decides who hears about it, how often, and what gets suppressed. In this tutorial you will install Prometheus and Alertmanager on Ubuntu 24.04, write two alerting rules, and build a routing tree that sends critical alerts and warnings to different receivers, with inhibition rules and silences managed through amtool.

Prerequisites

To follow this tutorial you need:

  • A server running Ubuntu 24.04 LTS, for example a CubePath VPS, with a non-root sudo user.
  • An SMTP account that accepts authenticated submission on port 587 (your mail provider or a transactional email service) if you want email notifications.
  • A Slack incoming webhook URL if you want Slack notifications. You can create one from a Slack app under Incoming Webhooks.

If Prometheus is already running on the server from the Ubuntu packages, you can skip the Prometheus part of Step 1.

Step 1 - Installing Prometheus and Alertmanager

Ubuntu 24.04 ships both components in its repositories, with systemd units and configuration under /etc/prometheus. The prometheus package also pulls in prometheus-node-exporter, which exposes the host metrics used by the example rules:

sudo apt update
sudo apt install -y prometheus prometheus-alertmanager

Check that the three services are running:

systemctl is-active prometheus prometheus-node-exporter prometheus-alertmanager

active
active
active

The packages include amtool, the Alertmanager command line client, and promtool, the Prometheus one. Confirm the versions:

amtool --version
promtool --version

By default Alertmanager listens on all interfaces on port 9093 (API) and 9094 (cluster gossip), without authentication. On a single server you do not need clustering, and only Prometheus on the same host must reach the API. Open the defaults file for the service:

sudo nano /etc/default/prometheus-alertmanager

Set the ARGS line so Alertmanager binds to localhost and clustering is disabled:

ARGS="--web.listen-address=127.0.0.1:9093 --cluster.listen-address="

Restart the service and confirm the listening address:

sudo systemctl restart prometheus-alertmanager
sudo ss -tlnp | grep 9093

LISTEN 0      4096       127.0.0.1:9093       0.0.0.0:*    users:(("prometheus-aler",pid=4121,fd=3))

To save typing, tell amtool where Alertmanager is:

mkdir -p ~/.config/amtool
echo "alertmanager.url: http://127.0.0.1:9093" > ~/.config/amtool/config.yml

Step 2 - Connecting Prometheus to Alertmanager

Prometheus needs two things: the address of Alertmanager and a set of alerting rules. Open the Prometheus configuration:

sudo nano /etc/prometheus/prometheus.yml

Make sure the alerting and rule_files sections look like this (the Ubuntu default already points to localhost:9093, you mainly need to add the rules path):

alerting:
  alertmanagers:
    - static_configs:
        - targets: ['localhost:9093']

rule_files:
  - /etc/prometheus/rules/*.yml

Create the rules directory and a rules file:

sudo mkdir -p /etc/prometheus/rules
sudo nano /etc/prometheus/rules/node.yml

The two rules below fire when a scrape target disappears and when the root filesystem is almost full. The severity label is what Alertmanager will route on:

groups:
  - name: node
    rules:
      - alert: InstanceDown
        expr: up == 0
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: "{{ $labels.instance }} is down"
          description: "Job {{ $labels.job }} on {{ $labels.instance }} has been unreachable for 2 minutes."

      - alert: RootDiskAlmostFull
        expr: (node_filesystem_avail_bytes{mountpoint="/",fstype!="tmpfs"} / node_filesystem_size_bytes{mountpoint="/",fstype!="tmpfs"}) * 100 < 15
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "Root disk on {{ $labels.instance }} below 15% free"
          description: "Only {{ printf \"%.1f\" $value }}% of / is free on {{ $labels.instance }}."

Validate both files with promtool before restarting:

promtool check rules /etc/prometheus/rules/node.yml
promtool check config /etc/prometheus/prometheus.yml

Checking /etc/prometheus/rules/node.yml
  SUCCESS: 2 rules found

Restart Prometheus and confirm it discovered Alertmanager:

sudo systemctl restart prometheus
curl -s http://localhost:9090/api/v1/alertmanagers

{"status":"success","data":{"activeAlertmanagers":[{"url":"http://localhost:9093/api/v2/alerts"}],"droppedAlertmanagers":[]}}

Step 3 - Understanding the Alertmanager configuration

The configuration file is /etc/prometheus/alertmanager.yml and has four main parts:

SectionPurpose
globalDefaults shared by receivers, such as SMTP settings and resolve_timeout.
routeA tree. The root route catches every alert; child routes match on labels and pick a receiver.
receiversNamed notification targets, each with one or more integrations (email, Slack, webhook...).
inhibit_rulesMute some alerts while other, more important alerts are firing.

Three timers on each route control how noisy notifications are:

  • group_wait: how long to wait after the first alert of a new group before notifying, so alerts that fire together arrive in one message.
  • group_interval: how long to wait before notifying about new alerts added to a group that was already notified.
  • repeat_interval: how long before re-sending a notification for alerts that are still firing.

Alerts are grouped by the labels listed in group_by. Grouping by alertname and instance means one message per problem per server.

Step 4 - Writing a routing configuration

Back up the sample file that the package installed:

sudo cp /etc/prometheus/alertmanager.yml /etc/prometheus/alertmanager.yml.orig
sudo nano /etc/prometheus/alertmanager.yml

Replace its content with the configuration below. Replace smtp.your_provider.com, the email addresses, your_smtp_password and the Slack webhook URL with your own values:

global:
  resolve_timeout: 5m
  smtp_smarthost: 'smtp.your_provider.com:587'
  smtp_from: 'alertmanager@your_domain'
  smtp_auth_username: 'alertmanager@your_domain'
  smtp_auth_password: 'your_smtp_password'
  smtp_require_tls: true

route:
  receiver: 'email-ops'
  group_by: ['alertname', 'instance']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
  routes:
    - matchers:
        - severity="critical"
      receiver: 'critical'
      group_wait: 10s
      repeat_interval: 1h
    - matchers:
        - severity="warning"
      receiver: 'slack-warnings'

receivers:
  - name: 'email-ops'
    email_configs:
      - to: 'ops@your_domain'
        send_resolved: true

  - name: 'critical'
    email_configs:
      - to: 'oncall@your_domain'
        send_resolved: true
    slack_configs:
      - api_url: 'https://hooks.slack.com/services/XXX/YYY/ZZZ'
        channel: '#alerts-critical'
        send_resolved: true
        title: '[{{ .Status | toUpper }}] {{ .CommonLabels.alertname }}'
        text: '{{ range .Alerts }}{{ .Annotations.description }}{{ "\n" }}{{ end }}'

  - name: 'slack-warnings'
    slack_configs:
      - api_url: 'https://hooks.slack.com/services/XXX/YYY/ZZZ'
        channel: '#alerts'
        send_resolved: true
        title: '[{{ .Status | toUpper }}] {{ .CommonLabels.alertname }}'
        text: '{{ range .Alerts }}{{ .Annotations.description }}{{ "\n" }}{{ end }}'

inhibit_rules:
  - source_matchers:
      - severity="critical"
    target_matchers:
      - severity="warning"
    equal: ['instance']

How this configuration behaves:

  • Critical alerts go to both the on-call mailbox and #alerts-critical, are sent 10 seconds after they fire, and repeat every hour while unresolved.
  • Warnings go only to #alerts, and repeat every 4 hours (inherited from the root route).
  • Anything without a matching severity falls through to the root receiver, email-ops.
  • While a critical alert fires for an instance, warnings for that same instance are muted, so a server that is down does not also flood the channel with disk or load warnings.

The matchers syntax replaces the older match and match_re keys, which are deprecated. It also accepts !=, =~ and !~, for example - service=~"api-.*".

The configuration contains a password, so restrict read access to root and the service user:

sudo chown root:prometheus /etc/prometheus/alertmanager.yml
sudo chmod 640 /etc/prometheus/alertmanager.yml

Step 5 - Validating and applying the configuration

Check the file with amtool. It validates the YAML, the templates and the receiver references:

amtool check-config /etc/prometheus/alertmanager.yml

Checking '/etc/prometheus/alertmanager.yml'  SUCCESS
Found:
 - global config
 - route
 - 1 inhibit rules
 - 3 receivers
 - 0 templates

Before going live, test which receiver a given set of labels would reach. This is the fastest way to debug a routing tree:

amtool config routes test --config.file=/etc/prometheus/alertmanager.yml severity=critical alertname=InstanceDown
amtool config routes test --config.file=/etc/prometheus/alertmanager.yml severity=info

critical
email-ops

Print the whole tree to review it:

amtool config routes show --config.file=/etc/prometheus/alertmanager.yml

Apply the new configuration. Alertmanager reloads without restarting when it receives a POST on /-/reload; if the file is invalid it keeps the previous configuration and logs the error:

curl -X POST http://127.0.0.1:9093/-/reload
sudo journalctl -u prometheus-alertmanager -n 20 --no-pager

... msg="Loading configuration file" file=/etc/prometheus/alertmanager.yml
... msg="Completed loading of configuration file" file=/etc/prometheus/alertmanager.yml

Step 6 - Sending a test alert

You do not need to break anything to test delivery. amtool alert add posts an alert directly to Alertmanager's API:

amtool alert add alertname=TestAlert severity=warning instance=test01 --annotation=description="Test warning sent with amtool"

List active alerts:

amtool alert query

Alertname  Starts At                Summary  State
TestAlert  2026-09-25 10:12:04 UTC           active

After group_wait (30 seconds) the message should appear in #alerts. Repeat with severity=critical to test the email and critical channel. Test alerts resolve on their own after resolve_timeout (5 minutes), and you will then receive the resolved notification because send_resolved is enabled.

To test the real pipeline from Prometheus, stop the node exporter for a couple of minutes, which makes up == 0 true for that target:

sudo systemctl stop prometheus-node-exporter

After the 2 minute for period, InstanceDown fires with severity=critical. Start the exporter again:

sudo systemctl start prometheus-node-exporter

Step 7 - Managing silences with amtool

A silence mutes notifications for alerts matching a set of labels during a time window, which is what you want during planned maintenance. Unlike inhibition, silences are created at runtime and do not require a configuration change.

Silence every alert from one instance for two hours:

amtool silence add instance="web01:9100" --duration=2h --comment="Kernel upgrade on web01"

The command prints the silence ID. List active silences:

amtool silence query

ID                                    Matchers              Ends At                  Created By  Comment
3f1a7c2e-8b4d-4e0f-9a51-2c6d7e8f9a01  instance="web01:9100"  2026-09-25 12:15:00 UTC  your_user   Kernel upgrade on web01

End a silence early when the maintenance is done:

amtool silence expire 3f1a7c2e-8b4d-4e0f-9a51-2c6d7e8f9a01

Silences are stored in Alertmanager's data directory and survive restarts.

Step 8 - Sending alerts to a webhook

For integrations that Alertmanager does not support natively, a webhook receiver POSTs a JSON document describing the alert group to any HTTP endpoint. Add a receiver like this to receivers:

  - name: 'ticketing'
    webhook_configs:
      - url: 'https://hooks.your_domain/alertmanager'
        send_resolved: true
        http_config:
          authorization:
            type: Bearer
            credentials: 'your_webhook_token'

Then reference it from a child route under route.routes, for example to send everything labeled team="billing" to it:

    - matchers:
        - team="billing"
      receiver: 'ticketing'

The payload contains status, groupLabels, commonLabels, commonAnnotations and the list of alerts. Validate and reload as in Step 5.

Troubleshooting

No notification arrives. Check the logs first; delivery errors (SMTP authentication, Slack 404) are logged with the receiver name:

sudo journalctl -u prometheus-alertmanager -f

Alerts fire in Prometheus but never reach Alertmanager. Query curl -s http://localhost:9090/api/v1/alertmanagers and confirm activeAlertmanagers is not empty. If you bound Alertmanager to 127.0.0.1, the target in prometheus.yml must be localhost:9093, not the public IP.

Email fails with 535 Authentication failed or TLS errors. Use port 587 with smtp_require_tls: true, and an app password if your provider requires it. Port 25 is often blocked outbound on cloud servers.

A notification went to the wrong receiver. Routes are evaluated top to bottom and the first match wins, unless a route sets continue: true. Reproduce the case with amtool config routes test and the exact labels of the alert.

amtool returns connection refused. Check ~/.config/amtool/config.yml points to the address set in --web.listen-address.

Conclusion

You now have Prometheus sending alerts to Alertmanager, which groups them by alert and instance, routes critical alerts and warnings to different channels, suppresses warnings while a critical alert is firing, and lets you silence noise during maintenance.

Good next steps are adding more rules (memory, systemd unit failures, certificate expiry with the blackbox exporter), moving repeated notification text into a template file referenced with templates:, and adding more Prometheus targets so the same routing tree covers your whole fleet.