--- title: "How to Set Up Prometheus, Grafana and Alertmanager on Ubuntu" description: "Build a monitoring stack on Ubuntu 24.04 with Prometheus, Node Exporter, Alertmanager and Grafana, including alert rules, email alerts and dashboards." category: "Observability Stack" language: en source: https://cubepath.com/docs/observability-stack/prometheus-grafana-complete-monitoring-stack updated: 2026-09-25 --- # How to Set Up Prometheus, Grafana and Alertmanager on Ubuntu A complete metrics stack needs four pieces: exporters that expose metrics, a server that collects and evaluates them, something that routes alerts to people, and a UI for dashboards. In this tutorial you will build that stack on Ubuntu 24.04 with Node Exporter, Prometheus, Alertmanager and Grafana, monitor both the monitoring server and additional hosts, define alert rules for down hosts, CPU, memory and disks, and receive notifications by email. ## Prerequisites To follow this tutorial you need: - A monitoring server running Ubuntu 24.04 LTS, for example a CubePath VPS, with at least 2 vCPUs, 4 GB of RAM and 40 GB of disk. - One or more servers to monitor, running Ubuntu 24.04 or Debian 12, reachable from the monitoring server over a private network or the internet. - A non-root user with `sudo` privileges on every server, and UFW enabled with SSH allowed. - An SMTP account to send alert emails (host, port, username and password). - The public IP address of your workstation, referred to as `your_admin_ip`. ## How the components fit together | Component | Port | Role | |---|---|---| | Node Exporter | 9100 | Runs on every host and exposes CPU, memory, disk and network metrics. | | Prometheus | 9090 | Scrapes exporters every 15 seconds, stores the data and evaluates alert rules. | | Alertmanager | 9093 | Receives firing alerts from Prometheus, groups and deduplicates them, and sends notifications. | | Grafana | 3000 | Queries Prometheus and shows dashboards. | Prometheus pulls from exporters; alerts flow from Prometheus to Alertmanager; Grafana only reads. In this setup Prometheus and Alertmanager listen on `127.0.0.1` only, and Grafana is the only service reachable from your workstation. ## Step 1 - Installing Node Exporter on every host Ubuntu and Debian package Node Exporter with a systemd service and security updates. Run this on the monitoring server **and** on each host you want to monitor: ```bash sudo apt update sudo apt install prometheus-node-exporter ``` Verify that it answers locally: ```bash curl -s http://localhost:9100/metrics | grep '^node_uname_info' ``` ```output node_uname_info{domainname="(none)",machine="x86_64",nodename="web-01",release="6.8.0-79-generic",sysname="Linux",version="#79-Ubuntu SMP ..."} 1 ``` Node Exporter has no authentication. On each monitored host (not the monitoring server), allow port 9100 only from the monitoring server, replacing `your_monitoring_ip` with its address: ```bash sudo ufw allow from your_monitoring_ip to any port 9100 proto tcp ``` From the monitoring server, confirm you can reach each host: ```bash curl -s http://web01_ip:9100/metrics | head -n 3 ``` If the command hangs, check the firewall rule on the host and any cloud firewall in front of it. ## Step 2 - Installing Prometheus The rest of the steps run on the monitoring server. Create a system user and the directories Prometheus needs: ```bash sudo useradd --system --no-create-home --shell /usr/sbin/nologin prometheus sudo mkdir -p /etc/prometheus/rules /var/lib/prometheus sudo chown prometheus:prometheus /var/lib/prometheus ``` Download the latest release from the [Prometheus download page](https://prometheus.io/download/) and verify its checksum. Adjust the version to the current one: ```bash PROM_VERSION=3.15.0 cd /tmp curl -LO "https://github.com/prometheus/prometheus/releases/download/v${PROM_VERSION}/prometheus-${PROM_VERSION}.linux-amd64.tar.gz" curl -Lo prometheus-sha256sums.txt "https://github.com/prometheus/prometheus/releases/download/v${PROM_VERSION}/sha256sums.txt" sha256sum --check --ignore-missing prometheus-sha256sums.txt ``` ```output prometheus-3.15.0.linux-amd64.tar.gz: OK ``` Install the binaries: ```bash tar xzf "prometheus-${PROM_VERSION}.linux-amd64.tar.gz" sudo install -m 0755 "prometheus-${PROM_VERSION}.linux-amd64/prometheus" "prometheus-${PROM_VERSION}.linux-amd64/promtool" /usr/local/bin/ prometheus --version | head -n 1 ``` ```output prometheus, version 3.15.0 (branch: HEAD, revision: ...) ``` You will write its configuration in Step 5, after Alertmanager and the alert rules exist. ## Step 3 - Installing Alertmanager Alertmanager is a separate binary with its own user and data directory: ```bash sudo useradd --system --no-create-home --shell /usr/sbin/nologin alertmanager sudo mkdir -p /etc/alertmanager /var/lib/alertmanager sudo chown alertmanager:alertmanager /var/lib/alertmanager ``` Download and verify it, using the current version from the same download page: ```bash AM_VERSION=0.34.1 cd /tmp curl -LO "https://github.com/prometheus/alertmanager/releases/download/v${AM_VERSION}/alertmanager-${AM_VERSION}.linux-amd64.tar.gz" curl -Lo alertmanager-sha256sums.txt "https://github.com/prometheus/alertmanager/releases/download/v${AM_VERSION}/sha256sums.txt" sha256sum --check --ignore-missing alertmanager-sha256sums.txt tar xzf "alertmanager-${AM_VERSION}.linux-amd64.tar.gz" sudo install -m 0755 "alertmanager-${AM_VERSION}.linux-amd64/alertmanager" "alertmanager-${AM_VERSION}.linux-amd64/amtool" /usr/local/bin/ ``` Create the configuration. It sends every alert by email, groups alerts that share an `alertname` and `instance` into one message, and repeats unresolved alerts every 4 hours. Replace the SMTP values and addresses with your own: ```bash sudo nano /etc/alertmanager/alertmanager.yml ``` ```yaml global: smtp_smarthost: "smtp.example.com:587" smtp_from: "alertmanager@example.com" smtp_auth_username: "alertmanager@example.com" smtp_auth_password: "your_smtp_password" smtp_require_tls: true route: receiver: ops-email group_by: ["alertname", "instance"] group_wait: 30s group_interval: 5m repeat_interval: 4h receivers: - name: ops-email email_configs: - to: "ops@example.com" send_resolved: true inhibit_rules: - source_matchers: ['severity="critical"'] target_matchers: ['severity="warning"'] equal: ["alertname", "instance"] ``` The `inhibit_rules` block silences a `warning` while a `critical` alert with the same name is firing on the same host, so you do not get two emails about one problem. The file contains the SMTP password, so restrict it and validate it: ```bash sudo chown root:alertmanager /etc/alertmanager/alertmanager.yml sudo chmod 640 /etc/alertmanager/alertmanager.yml sudo amtool check-config /etc/alertmanager/alertmanager.yml ``` ```output Checking '/etc/alertmanager/alertmanager.yml' SUCCESS Found: - global config - route - 1 inhibit rules - 1 receivers - 0 templates ``` Create the systemd unit. `--cluster.listen-address=` with an empty value disables the high-availability gossip port, which a single instance does not need: ```bash sudo nano /etc/systemd/system/alertmanager.service ``` ```ini [Unit] Description=Prometheus Alertmanager Wants=network-online.target After=network-online.target [Service] User=alertmanager Group=alertmanager Type=simple ExecStart=/usr/local/bin/alertmanager \ --config.file=/etc/alertmanager/alertmanager.yml \ --storage.path=/var/lib/alertmanager \ --web.listen-address=127.0.0.1:9093 \ --cluster.listen-address= ExecReload=/bin/kill -HUP $MAINPID Restart=on-failure NoNewPrivileges=true ProtectSystem=full ProtectHome=true [Install] WantedBy=multi-user.target ``` Start it and check that it is ready: ```bash sudo systemctl daemon-reload sudo systemctl enable --now alertmanager curl -s http://127.0.0.1:9093/-/ready ``` ```output OK ``` ## Step 4 - Writing alert rules Alert rules are PromQL expressions that Prometheus evaluates on a schedule. When an expression returns results for longer than `for`, the alert fires and is sent to Alertmanager. Create a rules file for host alerts: ```bash sudo nano /etc/prometheus/rules/node.yml ``` ```yaml groups: - name: node rules: - alert: InstanceDown expr: up == 0 for: 2m labels: severity: critical annotations: summary: "{{ $labels.instance }} is down" description: "Prometheus has not been able to scrape {{ $labels.instance }} (job {{ $labels.job }}) for 2 minutes." - alert: HostHighCpu expr: 100 * (1 - avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[5m]))) > 90 for: 10m labels: severity: warning annotations: summary: "High CPU on {{ $labels.instance }}" description: "CPU usage has been above 90% for 10 minutes (current: {{ $value | printf \"%.1f\" }}%)." - alert: HostLowMemory expr: 100 * (1 - node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes) > 90 for: 5m labels: severity: warning annotations: summary: "Low memory on {{ $labels.instance }}" description: "Memory usage is {{ $value | printf \"%.1f\" }}%." - alert: HostDiskAlmostFull expr: 100 * node_filesystem_avail_bytes{fstype!~"tmpfs|overlay"} / node_filesystem_size_bytes{fstype!~"tmpfs|overlay"} < 10 for: 5m labels: severity: warning annotations: summary: "Disk almost full on {{ $labels.instance }}" description: "{{ $labels.mountpoint }} has {{ $value | printf \"%.1f\" }}% free space left." - alert: HostDiskWillFillIn4Hours expr: predict_linear(node_filesystem_avail_bytes{fstype!~"tmpfs|overlay"}[1h], 4 * 3600) < 0 for: 10m labels: severity: critical annotations: summary: "Disk on {{ $labels.instance }} will fill within 4 hours" description: "At the current rate, {{ $labels.mountpoint }} will run out of space in less than 4 hours." ``` The `severity` label is what the inhibit rule in Alertmanager matches on, and the annotations become the body of the email. Validate the file: ```bash promtool check rules /etc/prometheus/rules/node.yml ``` ```output Checking /etc/prometheus/rules/node.yml SUCCESS: 5 rules found ``` ## Step 5 - Configuring and starting Prometheus Now write the Prometheus configuration. It scrapes Prometheus, Alertmanager and every Node Exporter, loads the rules, and sends alerts to Alertmanager. Replace the example IPs and names with your hosts: ```bash sudo nano /etc/prometheus/prometheus.yml ``` ```yaml global: scrape_interval: 15s evaluation_interval: 15s alerting: alertmanagers: - static_configs: - targets: ["127.0.0.1:9093"] rule_files: - /etc/prometheus/rules/*.yml scrape_configs: - job_name: prometheus static_configs: - targets: ["127.0.0.1:9090"] - job_name: alertmanager static_configs: - targets: ["127.0.0.1:9093"] - job_name: node static_configs: - targets: ["127.0.0.1:9100"] labels: host: monitoring-01 - targets: ["10.0.0.11:9100"] labels: host: web-01 - targets: ["10.0.0.12:9100"] labels: host: db-01 ``` Validate the configuration. `promtool` also checks every rules file it references: ```bash promtool check config /etc/prometheus/prometheus.yml ``` ```output Checking /etc/prometheus/prometheus.yml SUCCESS: 1 rule files found SUCCESS: /etc/prometheus/prometheus.yml is valid prometheus config file syntax Checking /etc/prometheus/rules/node.yml SUCCESS: 5 rules found ``` Create the systemd unit. Prometheus keeps 30 days of data or 20 GB, whichever limit is reached first: ```bash sudo nano /etc/systemd/system/prometheus.service ``` ```ini [Unit] Description=Prometheus monitoring system Wants=network-online.target After=network-online.target [Service] User=prometheus Group=prometheus Type=simple ExecStart=/usr/local/bin/prometheus \ --config.file=/etc/prometheus/prometheus.yml \ --storage.tsdb.path=/var/lib/prometheus \ --storage.tsdb.retention.time=30d \ --storage.tsdb.retention.size=20GB \ --web.listen-address=127.0.0.1:9090 ExecReload=/bin/kill -HUP $MAINPID Restart=on-failure NoNewPrivileges=true ProtectSystem=full ProtectHome=true [Install] WantedBy=multi-user.target ``` Start Prometheus: ```bash sudo systemctl daemon-reload sudo systemctl enable --now prometheus curl -s http://127.0.0.1:9090/-/ready ``` ```output Prometheus Server is Ready. ``` Check that every target is up. The `up` metric is `1` for each target whose last scrape succeeded, so this query should return the number of targets you configured (5 in this example): ```bash curl -s 'http://127.0.0.1:9090/api/v1/query?query=count(up==1)' ``` ```output {"status":"success","data":{"resultType":"vector","result":[{"metric":{},"value":[1790336120.456,"5"]}]}} ``` You can also use the web UI. Because Prometheus listens on localhost only, open an SSH tunnel from your workstation: ```bash ssh -L 9090:127.0.0.1:9090 -L 9093:127.0.0.1:9093 your_user@your_server_ip ``` With the tunnel open, browse to `http://localhost:9090/targets` and confirm every target is **UP**. The **Alerts** page lists the five rules, all **Inactive**. `http://localhost:9093` shows the Alertmanager UI. ## Step 6 - Installing Grafana Add the Grafana APT repository and install Grafana OSS: ```bash sudo apt install apt-transport-https wget sudo mkdir -p /etc/apt/keyrings sudo wget -O /etc/apt/keyrings/grafana.asc https://apt.grafana.com/gpg-full.key sudo chmod 644 /etc/apt/keyrings/grafana.asc echo "deb [signed-by=/etc/apt/keyrings/grafana.asc] https://apt.grafana.com stable main" | sudo tee /etc/apt/sources.list.d/grafana.list sudo apt update sudo apt install grafana ``` Provision Prometheus and Alertmanager as data sources so they exist the moment Grafana starts: ```bash sudo nano /etc/grafana/provisioning/datasources/stack.yaml ``` ```yaml apiVersion: 1 datasources: - name: Prometheus uid: prometheus type: prometheus access: proxy url: http://127.0.0.1:9090 isDefault: true jsonData: timeInterval: 15s - name: Alertmanager uid: alertmanager type: alertmanager access: proxy url: http://127.0.0.1:9093 jsonData: implementation: prometheus ``` Start Grafana and allow port 3000 from your workstation only: ```bash sudo systemctl daemon-reload sudo systemctl enable --now grafana-server sudo ufw allow from your_admin_ip to any port 3000 proto tcp ``` Browse to `http://your_server_ip:3000`, log in with `admin` / `admin`, and set a new password when prompted. Go to **Connections > Data sources**, open **Prometheus** and click **Test**; you should see `Successfully queried the Prometheus API.` > Important: Plain HTTP sends your Grafana password unencrypted. For anything beyond a test, put Grafana behind Nginx with a Let's Encrypt certificate and set `root_url` in `/etc/grafana/grafana.ini`, as described in the tutorial "How to Install Grafana on Ubuntu 24.04 and Build Dashboards". ## Step 7 - Importing dashboards Instead of building host dashboards from scratch, import the community standard: 1. Go to **Dashboards > New > Import**. 2. Enter `1860` (**Node Exporter Full**) and click **Load**. 3. Select the **Prometheus** data source and click **Import**. Use the **Host** drop-down at the top to switch between `monitoring-01`, `web-01` and `db-01`. The CPU, memory, disk and network panels should all show data within a minute. Because Alertmanager is also a data source, **Alerting > Alert rules** in Grafana lists the Prometheus rules, and **Alerting > Silences** (with the Alertmanager data source selected) lets you create silences for planned maintenance without leaving Grafana. ## Step 8 - Testing the alerting pipeline Test the path in two stages. First, confirm that Alertmanager can deliver email by injecting a fake alert with `amtool`: ```bash amtool alert add alertname=TestAlert severity=warning instance=test-host \ --annotation=summary="Test alert from amtool" \ --alertmanager.url=http://127.0.0.1:9093 ``` After the 30-second `group_wait`, the message arrives at the address in `alertmanager.yml`. If it does not, the Alertmanager log shows the SMTP error: ```bash sudo journalctl -u alertmanager -n 30 --no-pager ``` Next, test a real alert end to end. On one of the monitored hosts, stop Node Exporter: ```bash sudo systemctl stop prometheus-node-exporter ``` Within about 15 seconds the target turns **DOWN** in Prometheus and `InstanceDown` becomes **Pending**. After the 2-minute `for` period it turns **Firing**, and you can list it from the monitoring server: ```bash amtool alert query --alertmanager.url=http://127.0.0.1:9093 ``` ```output Alertname Starts At Summary InstanceDown 2026-09-25 11:32:10 UTC 10.0.0.11:9100 is down ``` The email follows. Start the exporter again and, because `send_resolved` is `true`, a **RESOLVED** email arrives a few minutes later: ```bash sudo systemctl start prometheus-node-exporter ``` ## Step 9 - Adding more hosts later To monitor a new server, install `prometheus-node-exporter` on it and open port 9100 to the monitoring server as in Step 1. Then add its target to the `node` job in `/etc/prometheus/prometheus.yml` and reload Prometheus without a restart: ```bash promtool check config /etc/prometheus/prometheus.yml && sudo systemctl reload prometheus ``` With many hosts, editing `prometheus.yml` becomes tedious. Move the targets to a file and use file-based service discovery instead: replace `static_configs` in the `node` job with: ```yaml file_sd_configs: - files: - /etc/prometheus/targets/*.yml ``` Each file in `/etc/prometheus/targets/` holds a list of targets with labels, in the same shape as a `static_configs` entry, and Prometheus picks up changes automatically without a reload. ## Troubleshooting **Alerts show as firing in Prometheus but no email arrives.** Check `http://localhost:9090/status` (through the tunnel) under **Alertmanagers** for `127.0.0.1:9093`, then read `sudo journalctl -u alertmanager` for SMTP authentication or TLS errors. With port 587, keep `smtp_require_tls: true` and make sure the username and password are correct. **A remote target is `DOWN` with `context deadline exceeded`.** The monitoring server cannot reach port 9100. Check the UFW rule on the target and any provider firewall between the two hosts. **`HostDiskAlmostFull` fires for `/boot/efi` or snap mounts.** Exclude those mount points by adding `mountpoint!~"/boot/efi|/snap/.*"` to the selector in the rule. **Grafana panels show `No data`.** The Prometheus data source URL must be `http://127.0.0.1:9090`, since Prometheus listens only on localhost. Test the data source and check that the `node` job is `UP`. ## Conclusion You now have a working monitoring stack: Node Exporter on every host, Prometheus collecting and evaluating alert rules, Alertmanager sending grouped email notifications with resolution messages, and Grafana dashboards for day-to-day visibility. From here, add exporters for your services (Blackbox Exporter for HTTP and TLS checks, MySQL or PostgreSQL exporters), route critical alerts to Slack or PagerDuty with a second Alertmanager receiver, and back up `/etc/prometheus`, `/etc/alertmanager` and `/var/lib/grafana` regularly.