--- title: "How to Configure Alertmanager for Prometheus on Ubuntu 24.04" description: "Install Alertmanager on Ubuntu 24.04, connect it to Prometheus, and route alerts by severity to email, Slack and webhooks with inhibition rules and silences." category: "Observability Stack" language: en source: https://cubepath.com/docs/observability-stack/alertmanager-configuration-for-prometheus updated: 2026-09-25 --- # How to Configure Alertmanager for Prometheus on Ubuntu 24.04 Alertmanager receives the alerts that Prometheus fires, groups and deduplicates them, and delivers notifications to channels such as email, Slack or any webhook. Prometheus decides *when* something is wrong; Alertmanager decides *who* hears about it, *how often*, and *what gets suppressed*. In this tutorial you will install Prometheus and Alertmanager on Ubuntu 24.04, write two alerting rules, and build a routing tree that sends critical alerts and warnings to different receivers, with inhibition rules and silences managed through `amtool`. ## Prerequisites To follow this tutorial you need: - A server running Ubuntu 24.04 LTS, for example a CubePath VPS, with a non-root `sudo` user. - An SMTP account that accepts authenticated submission on port 587 (your mail provider or a transactional email service) if you want email notifications. - A Slack incoming webhook URL if you want Slack notifications. You can create one from a Slack app under **Incoming Webhooks**. If Prometheus is already running on the server from the Ubuntu packages, you can skip the Prometheus part of Step 1. ## Step 1 - Installing Prometheus and Alertmanager Ubuntu 24.04 ships both components in its repositories, with systemd units and configuration under `/etc/prometheus`. The `prometheus` package also pulls in `prometheus-node-exporter`, which exposes the host metrics used by the example rules: ```bash sudo apt update sudo apt install -y prometheus prometheus-alertmanager ``` Check that the three services are running: ```bash systemctl is-active prometheus prometheus-node-exporter prometheus-alertmanager ``` ```output active active active ``` The packages include `amtool`, the Alertmanager command line client, and `promtool`, the Prometheus one. Confirm the versions: ```bash amtool --version promtool --version ``` By default Alertmanager listens on all interfaces on port 9093 (API) and 9094 (cluster gossip), without authentication. On a single server you do not need clustering, and only Prometheus on the same host must reach the API. Open the defaults file for the service: ```bash sudo nano /etc/default/prometheus-alertmanager ``` Set the `ARGS` line so Alertmanager binds to localhost and clustering is disabled: ```ini ARGS="--web.listen-address=127.0.0.1:9093 --cluster.listen-address=" ``` Restart the service and confirm the listening address: ```bash sudo systemctl restart prometheus-alertmanager sudo ss -tlnp | grep 9093 ``` ```output LISTEN 0 4096 127.0.0.1:9093 0.0.0.0:* users:(("prometheus-aler",pid=4121,fd=3)) ``` To save typing, tell `amtool` where Alertmanager is: ```bash mkdir -p ~/.config/amtool echo "alertmanager.url: http://127.0.0.1:9093" > ~/.config/amtool/config.yml ``` ## Step 2 - Connecting Prometheus to Alertmanager Prometheus needs two things: the address of Alertmanager and a set of alerting rules. Open the Prometheus configuration: ```bash sudo nano /etc/prometheus/prometheus.yml ``` Make sure the `alerting` and `rule_files` sections look like this (the Ubuntu default already points to `localhost:9093`, you mainly need to add the rules path): ```yaml alerting: alertmanagers: - static_configs: - targets: ['localhost:9093'] rule_files: - /etc/prometheus/rules/*.yml ``` Create the rules directory and a rules file: ```bash sudo mkdir -p /etc/prometheus/rules sudo nano /etc/prometheus/rules/node.yml ``` The two rules below fire when a scrape target disappears and when the root filesystem is almost full. The `severity` label is what Alertmanager will route on: ```yaml groups: - name: node rules: - alert: InstanceDown expr: up == 0 for: 2m labels: severity: critical annotations: summary: "{{ $labels.instance }} is down" description: "Job {{ $labels.job }} on {{ $labels.instance }} has been unreachable for 2 minutes." - alert: RootDiskAlmostFull expr: (node_filesystem_avail_bytes{mountpoint="/",fstype!="tmpfs"} / node_filesystem_size_bytes{mountpoint="/",fstype!="tmpfs"}) * 100 < 15 for: 10m labels: severity: warning annotations: summary: "Root disk on {{ $labels.instance }} below 15% free" description: "Only {{ printf \"%.1f\" $value }}% of / is free on {{ $labels.instance }}." ``` Validate both files with `promtool` before restarting: ```bash promtool check rules /etc/prometheus/rules/node.yml promtool check config /etc/prometheus/prometheus.yml ``` ```output Checking /etc/prometheus/rules/node.yml SUCCESS: 2 rules found ``` Restart Prometheus and confirm it discovered Alertmanager: ```bash sudo systemctl restart prometheus curl -s http://localhost:9090/api/v1/alertmanagers ``` ```output {"status":"success","data":{"activeAlertmanagers":[{"url":"http://localhost:9093/api/v2/alerts"}],"droppedAlertmanagers":[]}} ``` ## Step 3 - Understanding the Alertmanager configuration The configuration file is `/etc/prometheus/alertmanager.yml` and has four main parts: | Section | Purpose | |---|---| | `global` | Defaults shared by receivers, such as SMTP settings and `resolve_timeout`. | | `route` | A tree. The root route catches every alert; child routes match on labels and pick a receiver. | | `receivers` | Named notification targets, each with one or more integrations (email, Slack, webhook...). | | `inhibit_rules` | Mute some alerts while other, more important alerts are firing. | Three timers on each route control how noisy notifications are: - `group_wait`: how long to wait after the first alert of a new group before notifying, so alerts that fire together arrive in one message. - `group_interval`: how long to wait before notifying about new alerts added to a group that was already notified. - `repeat_interval`: how long before re-sending a notification for alerts that are still firing. Alerts are grouped by the labels listed in `group_by`. Grouping by `alertname` and `instance` means one message per problem per server. ## Step 4 - Writing a routing configuration Back up the sample file that the package installed: ```bash sudo cp /etc/prometheus/alertmanager.yml /etc/prometheus/alertmanager.yml.orig sudo nano /etc/prometheus/alertmanager.yml ``` Replace its content with the configuration below. Replace `smtp.your_provider.com`, the email addresses, `your_smtp_password` and the Slack webhook URL with your own values: ```yaml global: resolve_timeout: 5m smtp_smarthost: 'smtp.your_provider.com:587' smtp_from: 'alertmanager@your_domain' smtp_auth_username: 'alertmanager@your_domain' smtp_auth_password: 'your_smtp_password' smtp_require_tls: true route: receiver: 'email-ops' group_by: ['alertname', 'instance'] group_wait: 30s group_interval: 5m repeat_interval: 4h routes: - matchers: - severity="critical" receiver: 'critical' group_wait: 10s repeat_interval: 1h - matchers: - severity="warning" receiver: 'slack-warnings' receivers: - name: 'email-ops' email_configs: - to: 'ops@your_domain' send_resolved: true - name: 'critical' email_configs: - to: 'oncall@your_domain' send_resolved: true slack_configs: - api_url: 'https://hooks.slack.com/services/XXX/YYY/ZZZ' channel: '#alerts-critical' send_resolved: true title: '[{{ .Status | toUpper }}] {{ .CommonLabels.alertname }}' text: '{{ range .Alerts }}{{ .Annotations.description }}{{ "\n" }}{{ end }}' - name: 'slack-warnings' slack_configs: - api_url: 'https://hooks.slack.com/services/XXX/YYY/ZZZ' channel: '#alerts' send_resolved: true title: '[{{ .Status | toUpper }}] {{ .CommonLabels.alertname }}' text: '{{ range .Alerts }}{{ .Annotations.description }}{{ "\n" }}{{ end }}' inhibit_rules: - source_matchers: - severity="critical" target_matchers: - severity="warning" equal: ['instance'] ``` How this configuration behaves: - Critical alerts go to both the on-call mailbox and `#alerts-critical`, are sent 10 seconds after they fire, and repeat every hour while unresolved. - Warnings go only to `#alerts`, and repeat every 4 hours (inherited from the root route). - Anything without a matching `severity` falls through to the root receiver, `email-ops`. - While a critical alert fires for an instance, warnings for that same instance are muted, so a server that is down does not also flood the channel with disk or load warnings. The `matchers` syntax replaces the older `match` and `match_re` keys, which are deprecated. It also accepts `!=`, `=~` and `!~`, for example `- service=~"api-.*"`. The configuration contains a password, so restrict read access to root and the service user: ```bash sudo chown root:prometheus /etc/prometheus/alertmanager.yml sudo chmod 640 /etc/prometheus/alertmanager.yml ``` ## Step 5 - Validating and applying the configuration Check the file with `amtool`. It validates the YAML, the templates and the receiver references: ```bash amtool check-config /etc/prometheus/alertmanager.yml ``` ```output Checking '/etc/prometheus/alertmanager.yml' SUCCESS Found: - global config - route - 1 inhibit rules - 3 receivers - 0 templates ``` Before going live, test which receiver a given set of labels would reach. This is the fastest way to debug a routing tree: ```bash amtool config routes test --config.file=/etc/prometheus/alertmanager.yml severity=critical alertname=InstanceDown amtool config routes test --config.file=/etc/prometheus/alertmanager.yml severity=info ``` ```output critical email-ops ``` Print the whole tree to review it: ```bash amtool config routes show --config.file=/etc/prometheus/alertmanager.yml ``` Apply the new configuration. Alertmanager reloads without restarting when it receives a POST on `/-/reload`; if the file is invalid it keeps the previous configuration and logs the error: ```bash curl -X POST http://127.0.0.1:9093/-/reload sudo journalctl -u prometheus-alertmanager -n 20 --no-pager ``` ```output ... msg="Loading configuration file" file=/etc/prometheus/alertmanager.yml ... msg="Completed loading of configuration file" file=/etc/prometheus/alertmanager.yml ``` ## Step 6 - Sending a test alert You do not need to break anything to test delivery. `amtool alert add` posts an alert directly to Alertmanager's API: ```bash amtool alert add alertname=TestAlert severity=warning instance=test01 --annotation=description="Test warning sent with amtool" ``` List active alerts: ```bash amtool alert query ``` ```output Alertname Starts At Summary State TestAlert 2026-09-25 10:12:04 UTC active ``` After `group_wait` (30 seconds) the message should appear in `#alerts`. Repeat with `severity=critical` to test the email and critical channel. Test alerts resolve on their own after `resolve_timeout` (5 minutes), and you will then receive the resolved notification because `send_resolved` is enabled. To test the real pipeline from Prometheus, stop the node exporter for a couple of minutes, which makes `up == 0` true for that target: ```bash sudo systemctl stop prometheus-node-exporter ``` After the 2 minute `for` period, `InstanceDown` fires with `severity=critical`. Start the exporter again: ```bash sudo systemctl start prometheus-node-exporter ``` ## Step 7 - Managing silences with amtool A silence mutes notifications for alerts matching a set of labels during a time window, which is what you want during planned maintenance. Unlike inhibition, silences are created at runtime and do not require a configuration change. Silence every alert from one instance for two hours: ```bash amtool silence add instance="web01:9100" --duration=2h --comment="Kernel upgrade on web01" ``` The command prints the silence ID. List active silences: ```bash amtool silence query ``` ```output ID Matchers Ends At Created By Comment 3f1a7c2e-8b4d-4e0f-9a51-2c6d7e8f9a01 instance="web01:9100" 2026-09-25 12:15:00 UTC your_user Kernel upgrade on web01 ``` End a silence early when the maintenance is done: ```bash amtool silence expire 3f1a7c2e-8b4d-4e0f-9a51-2c6d7e8f9a01 ``` Silences are stored in Alertmanager's data directory and survive restarts. ## Step 8 - Sending alerts to a webhook For integrations that Alertmanager does not support natively, a webhook receiver POSTs a JSON document describing the alert group to any HTTP endpoint. Add a receiver like this to `receivers`: ```yaml - name: 'ticketing' webhook_configs: - url: 'https://hooks.your_domain/alertmanager' send_resolved: true http_config: authorization: type: Bearer credentials: 'your_webhook_token' ``` Then reference it from a child route under `route.routes`, for example to send everything labeled `team="billing"` to it: ```yaml - matchers: - team="billing" receiver: 'ticketing' ``` The payload contains `status`, `groupLabels`, `commonLabels`, `commonAnnotations` and the list of `alerts`. Validate and reload as in Step 5. ## Troubleshooting **No notification arrives.** Check the logs first; delivery errors (SMTP authentication, Slack 404) are logged with the receiver name: ```bash sudo journalctl -u prometheus-alertmanager -f ``` **Alerts fire in Prometheus but never reach Alertmanager.** Query `curl -s http://localhost:9090/api/v1/alertmanagers` and confirm `activeAlertmanagers` is not empty. If you bound Alertmanager to `127.0.0.1`, the target in `prometheus.yml` must be `localhost:9093`, not the public IP. **Email fails with `535 Authentication failed` or TLS errors.** Use port 587 with `smtp_require_tls: true`, and an app password if your provider requires it. Port 25 is often blocked outbound on cloud servers. **A notification went to the wrong receiver.** Routes are evaluated top to bottom and the first match wins, unless a route sets `continue: true`. Reproduce the case with `amtool config routes test` and the exact labels of the alert. **`amtool` returns `connection refused`.** Check `~/.config/amtool/config.yml` points to the address set in `--web.listen-address`. ## Conclusion You now have Prometheus sending alerts to Alertmanager, which groups them by alert and instance, routes critical alerts and warnings to different channels, suppresses warnings while a critical alert is firing, and lets you silence noise during maintenance. Good next steps are adding more rules (memory, systemd unit failures, certificate expiry with the blackbox exporter), moving repeated notification text into a template file referenced with `templates:`, and adding more Prometheus targets so the same routing tree covers your whole fleet.