Prometheus is an open source monitoring system that scrapes metrics over HTTP from your services at a fixed interval, stores them in a local time-series database and lets you query them with PromQL. In this tutorial you will install Prometheus 3 from the official release binaries on Ubuntu 24.04, run it as a systemd service under its own user, collect host metrics with Node Exporter, tune data retention, write your first PromQL queries and protect the web interface with a password.
Prerequisites
To follow this tutorial you need:
- A server running Ubuntu 24.04 LTS, for example a CubePath VPS, with at least 1 vCPU, 2 GB of RAM and 20 GB of free disk space. Storage needs grow with the number of series and the retention period.
- A non-root user with
sudoprivileges. - UFW enabled, with SSH allowed.
- The public IP address of the machine you will use to open the web UI, referred to as
your_admin_ip.
Step 1 - Creating the Prometheus user and directories
Prometheus should not run as root. Create a system user without a login shell or home directory:
sudo useradd --system --no-create-home --shell /usr/sbin/nologin prometheus
Create the configuration directory and the data directory, and give the data directory to the new user:
sudo mkdir -p /etc/prometheus /var/lib/prometheus
sudo chown prometheus:prometheus /var/lib/prometheus
The configuration stays owned by root so the Prometheus process can read it but not modify it.
Step 2 - Downloading and installing Prometheus
Ubuntu's own prometheus package lags far behind upstream, so this tutorial uses the official binaries. Check the Prometheus download page for the current release and set it in a variable:
PROM_VERSION=3.15.0
cd /tmp
curl -LO "https://github.com/prometheus/prometheus/releases/download/v${PROM_VERSION}/prometheus-${PROM_VERSION}.linux-amd64.tar.gz"
curl -LO "https://github.com/prometheus/prometheus/releases/download/v${PROM_VERSION}/sha256sums.txt"
On an arm64 server, replace linux-amd64 with linux-arm64 in the file name. Verify the archive against the published checksums:
sha256sum --check --ignore-missing sha256sums.txt
prometheus-3.15.0.linux-amd64.tar.gz: OK
Extract the archive and install the two binaries and the sample configuration:
tar xzf "prometheus-${PROM_VERSION}.linux-amd64.tar.gz"
cd "prometheus-${PROM_VERSION}.linux-amd64"
sudo install -m 0755 prometheus promtool /usr/local/bin/
sudo install -m 0644 prometheus.yml /etc/prometheus/prometheus.yml
prometheus is the server; promtool validates configuration files and runs queries from the command line. Confirm both are on your PATH:
prometheus --version
prometheus, version 3.15.0 (branch: HEAD, revision: ...)
build user: root@...
go version: go1.x
platform: linux/amd64
Step 3 - Installing Node Exporter
Prometheus only collects what something exposes. Node Exporter publishes host metrics (CPU, memory, disks, filesystems, network) on port 9100. Ubuntu packages it and runs it as a service with sensible defaults:
sudo apt update
sudo apt install prometheus-node-exporter
Check that the service is running and returning metrics:
systemctl status prometheus-node-exporter --no-pager
curl -s http://localhost:9100/metrics | grep '^node_load1'
node_load1 0.08
Install the same package on every other server you want to monitor. Port 9100 has no authentication, so on those servers allow it only from the Prometheus server:
sudo ufw allow from your_prometheus_ip to any port 9100 proto tcp
Step 4 - Writing the configuration
Replace the sample configuration with one that scrapes Prometheus itself and Node Exporter:
sudo nano /etc/prometheus/prometheus.yml
global:
scrape_interval: 15s
evaluation_interval: 15s
external_labels:
environment: production
scrape_configs:
- job_name: prometheus
static_configs:
- targets: ["localhost:9090"]
- job_name: node
static_configs:
- targets: ["localhost:9100"]
labels:
host: monitoring-01
# - targets: ["10.0.0.11:9100"]
# labels:
# host: web-01
The main keys are:
scrape_interval: how often every target is scraped. 15 seconds is a good default; lower values multiply storage use.evaluation_interval: how often recording and alerting rules are evaluated.external_labels: labels attached to every series when data leaves this server (remote write, federation, Alertmanager).scrape_configs: one entry per job. Each target gets ajoblabel with the job name and aninstancelabel with itshost:port. Extralabelslet you tag targets with friendlier names.
To add another server later, uncomment the second target block (or add a new one) with its IP address.
Validate the file before starting Prometheus. promtool catches indentation mistakes and unknown keys:
promtool check config /etc/prometheus/prometheus.yml
Checking /etc/prometheus/prometheus.yml
SUCCESS: /etc/prometheus/prometheus.yml is valid prometheus config file syntax
Step 5 - Creating the systemd service
Create a unit file so Prometheus starts at boot and restarts if it crashes:
sudo nano /etc/systemd/system/prometheus.service
[Unit]
Description=Prometheus monitoring system
Documentation=https://prometheus.io/docs/
Wants=network-online.target
After=network-online.target
[Service]
User=prometheus
Group=prometheus
Type=simple
ExecStart=/usr/local/bin/prometheus \
--config.file=/etc/prometheus/prometheus.yml \
--storage.tsdb.path=/var/lib/prometheus \
--storage.tsdb.retention.time=30d \
--storage.tsdb.retention.size=15GB \
--web.listen-address=0.0.0.0:9090
ExecReload=/bin/kill -HUP $MAINPID
Restart=on-failure
RestartSec=5s
NoNewPrivileges=true
ProtectSystem=full
ProtectHome=true
[Install]
WantedBy=multi-user.target
Retention is configured with two flags, and whichever limit is reached first wins:
--storage.tsdb.retention.time=30ddeletes data older than 30 days (the default is 15 days).--storage.tsdb.retention.size=15GBdeletes the oldest blocks when the database exceeds 15 GB. Set it to about 80% of the space you can give Prometheus, because the write-ahead log is not counted.
ExecReload sends SIGHUP, which makes Prometheus re-read its configuration without restarting or losing data.
Load the unit and start the service:
sudo systemctl daemon-reload
sudo systemctl enable --now prometheus
systemctl status prometheus --no-pager
● prometheus.service - Prometheus monitoring system
Loaded: loaded (/etc/systemd/system/prometheus.service; enabled; preset: enabled)
Active: active (running) since Thu 2026-09-25 10:20:14 UTC; 5s ago
Check the readiness endpoint and the logs:
curl -s http://localhost:9090/-/ready
sudo journalctl -u prometheus -n 20 --no-pager
Prometheus Server is Ready.
The log should end with Server is ready to receive web requests.
Step 6 - Opening the web UI and checking targets
Allow port 9090 only from your own IP. The UI and API are unauthenticated until you complete Step 8, so never open this port to everyone:
sudo ufw allow from your_admin_ip to any port 9090 proto tcp
Open http://your_server_ip:9090 in your browser and go to Status > Target health. Both the prometheus and node jobs should show their targets as UP. You can check the same thing from the shell through the HTTP API:
curl -s 'http://localhost:9090/api/v1/query?query=up'
{"status":"success","data":{"resultType":"vector","result":[{"metric":{"__name__":"up","instance":"localhost:9090","job":"prometheus"},"value":[1790331600.123,"1"]},{"metric":{"__name__":"up","host":"monitoring-01","instance":"localhost:9100","job":"node"},"value":[1790331600.123,"1"]}]}}
A value of 1 means the last scrape succeeded; 0 means the target is down.
Step 7 - Querying metrics with PromQL
Open the Query page in the UI and try the following expressions. Each one is a common building block for dashboards and alerts.
Show which targets are down:
up == 0
CPU usage per host, as a percentage. node_cpu_seconds_total is a counter, so rate() turns it into per-second values over the last 5 minutes, and the idle share is subtracted from 100:
100 * (1 - avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])))
Memory in use, as a percentage:
100 * (1 - node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)
Used space on each real filesystem, ignoring tmpfs and overlay mounts:
100 * (1 - node_filesystem_avail_bytes{fstype!~"tmpfs|overlay"} / node_filesystem_size_bytes{fstype!~"tmpfs|overlay"})
Inbound network traffic in bits per second, excluding the loopback interface:
rate(node_network_receive_bytes_total{device!="lo"}[5m]) * 8
Predict whether a filesystem will be full in the next 24 hours, based on the last 6 hours of growth:
predict_linear(node_filesystem_avail_bytes{fstype!~"tmpfs|overlay"}[6h], 24 * 3600) < 0
Two rules keep PromQL correct: always wrap counters (metrics ending in _total) in rate() or increase() before doing math on them, and use a range such as [5m] that is at least four times your scrape interval.
Step 8 - Protecting the web UI with basic authentication
Prometheus can require a username and password on its UI and API through a web configuration file. The password must be stored as a bcrypt hash; htpasswd from apache2-utils generates one:
sudo apt install apache2-utils
htpasswd -nBC 12 admin
Enter a strong password twice. The command prints a line like admin:$2y$12$.... Create the web configuration file and paste the hash after admin::
sudo nano /etc/prometheus/web.yml
basic_auth_users:
admin: $2y$12$replace_with_the_hash_printed_by_htpasswd
Validate it:
promtool check web-config /etc/prometheus/web.yml
/etc/prometheus/web.yml SUCCESS
Once authentication is on, Prometheus also needs the password to scrape its own /metrics endpoint. Store the plain-text password in a file only the prometheus group can read, replacing your_strong_password with the password you typed into htpasswd:
echo -n 'your_strong_password' | sudo tee /etc/prometheus/self_scrape_password > /dev/null
sudo chown root:prometheus /etc/prometheus/self_scrape_password
sudo chmod 0640 /etc/prometheus/self_scrape_password
Edit the prometheus job in /etc/prometheus/prometheus.yml so it sends those credentials:
sudo nano /etc/prometheus/prometheus.yml
- job_name: prometheus
basic_auth:
username: admin
password_file: /etc/prometheus/self_scrape_password
static_configs:
- targets: ["localhost:9090"]
Finally, add the web configuration flag to the service. Open the unit file:
sudo nano /etc/systemd/system/prometheus.service
Add --web.config.file to the ExecStart block so it looks like this:
ExecStart=/usr/local/bin/prometheus \
--config.file=/etc/prometheus/prometheus.yml \
--web.config.file=/etc/prometheus/web.yml \
--storage.tsdb.path=/var/lib/prometheus \
--storage.tsdb.retention.time=30d \
--storage.tsdb.retention.size=15GB \
--web.listen-address=0.0.0.0:9090
Validate the configuration, reload systemd and restart Prometheus:
promtool check config /etc/prometheus/prometheus.yml
sudo systemctl daemon-reload
sudo systemctl restart prometheus
Requests without credentials are now rejected, and requests with them succeed:
curl -s -o /dev/null -w '%{http_code}\n' 'http://localhost:9090/api/v1/query?query=up'
curl -s -u admin -o /dev/null -w '%{http_code}\n' 'http://localhost:9090/api/v1/query?query=up'
401
Enter host password for user 'admin':
200
Basic authentication sends the password on every request, so if you reach Prometheus over the internet, also put it behind HTTPS (for example an Nginx reverse proxy with a Let's Encrypt certificate) or keep port 9090 closed and use an SSH tunnel.
Step 9 - Reloading configuration and watching storage
Whenever you add targets, validate and reload instead of restarting:
promtool check config /etc/prometheus/prometheus.yml && sudo systemctl reload prometheus
Prometheus reports on itself, so you can watch the database from the Query page:
prometheus_tsdb_head_series
prometheus_tsdb_storage_blocks_bytes / 1024 / 1024 / 1024
The first query shows how many active series you store (the main driver of RAM use); the second shows the size of persisted blocks in GiB. Compare it with the disk itself:
sudo du -sh /var/lib/prometheus
Troubleshooting
Target shows DOWN with connection refused. The exporter is not listening or a firewall blocks the port. On the target, run curl -s localhost:9100/metrics | head and check sudo ufw status for a rule allowing the Prometheus server's IP.
Prometheus fails to start with permission denied on /var/lib/prometheus. The data directory is not owned by the service user. Run sudo chown -R prometheus:prometheus /var/lib/prometheus.
Reload does nothing. A reload with an invalid file is rejected and the old configuration keeps running. Check sudo journalctl -u prometheus -n 50 for Error reloading config and run promtool check config again.
The prometheus job turns DOWN with 401 Unauthorized after Step 8. The password in self_scrape_password does not match the hash in web.yml, or the file ends with a newline. Recreate it with echo -n.
Conclusion
You now have Prometheus 3 running as a hardened systemd service, scraping host metrics from Node Exporter, keeping 30 days of data within a fixed disk budget, and protected with a password. Next, connect Grafana to Prometheus to build dashboards, add alerting rules and Alertmanager to get notified when up == 0 or a disk is filling, and add exporters for the services you run (for example MySQL, Nginx or Blackbox Exporter for HTTP checks).
