--- title: "How to Monitor Endpoints with Prometheus Blackbox Exporter" description: "Install Prometheus Blackbox Exporter on Ubuntu 24.04, probe websites over HTTP, TCP, ICMP and DNS, and alert on downtime and TLS certificates close to expiry." category: "Observability Stack" language: en source: https://cubepath.com/docs/observability-stack/blackbox-exporter-for-endpoint-monitoring updated: 2026-09-25 --- # How to Monitor Endpoints with Prometheus Blackbox Exporter Blackbox Exporter is the official Prometheus exporter for black-box monitoring: instead of reading metrics from inside a service, it probes the service from the outside over HTTP, HTTPS, TCP, ICMP or DNS, the same way a user would. It tells you whether an endpoint answers, how long it took, which status code it returned and when its TLS certificate expires. In this tutorial you will install Blackbox Exporter on Ubuntu 24.04 next to Prometheus, define probe modules, scrape them with Prometheus and add alerts for downtime and certificate expiry. ## Prerequisites To follow this tutorial, you will need: - A server running Ubuntu 24.04 LTS, such as a CubePath VPS, with a non-root user with `sudo` privileges. - Prometheus installed on the same server with its configuration in `/etc/prometheus/prometheus.yml`. The Ubuntu package (`sudo apt install prometheus`) is enough for this guide. - One or more endpoints to probe, for example `https://your_domain`. The examples also use public targets so you can test without your own site. Blackbox Exporter only needs outbound access to the targets. It will listen on `127.0.0.1:9115`, so no inbound firewall rule is required. ## Step 1 - Installing Blackbox Exporter Ubuntu packages an older release of the exporter, so this guide uses the official binary from the project's GitHub releases. Check the latest version on the [releases page](https://github.com/prometheus/blackbox_exporter/releases) and set it in a shell variable: ```bash VERSION=0.27.0 cd /tmp curl -LO "https://github.com/prometheus/blackbox_exporter/releases/download/v${VERSION}/blackbox_exporter-${VERSION}.linux-amd64.tar.gz" curl -LO "https://github.com/prometheus/blackbox_exporter/releases/download/v${VERSION}/sha256sums.txt" ``` On an ARM server, replace `linux-amd64` with `linux-arm64`. Verify the checksum before extracting the archive: ```bash sha256sum -c --ignore-missing sha256sums.txt ``` ```output blackbox_exporter-0.27.0.linux-amd64.tar.gz: OK ``` Extract the archive and install the binary: ```bash tar xzf "blackbox_exporter-${VERSION}.linux-amd64.tar.gz" sudo install -m 0755 "blackbox_exporter-${VERSION}.linux-amd64/blackbox_exporter" /usr/local/bin/ ``` Create a dedicated system user without a login shell to run the service: ```bash sudo useradd --system --no-create-home --shell /usr/sbin/nologin blackbox ``` Confirm the binary works: ```bash blackbox_exporter --version ``` ```output blackbox_exporter, version 0.27.0 (branch: HEAD, revision: ...) ``` ## Step 2 - Defining probe modules A module describes how to probe a target: which prober to use, the timeout and what counts as success. Prometheus later chooses a module per scrape job and passes the target as a URL parameter. Create the configuration directory and file: ```bash sudo mkdir -p /etc/blackbox_exporter sudo nano /etc/blackbox_exporter/blackbox.yml ``` Add the following modules: ```yaml modules: http_2xx: prober: http timeout: 5s http: method: GET preferred_ip_protocol: ip4 follow_redirects: true http_keyword: prober: http timeout: 5s http: preferred_ip_protocol: ip4 fail_if_body_not_matches_regexp: - "Example Domain" tcp_connect: prober: tcp timeout: 5s icmp: prober: icmp timeout: 5s icmp: preferred_ip_protocol: ip4 dns_a: prober: dns timeout: 5s dns: query_name: example.com query_type: A valid_rcodes: - NOERROR ``` What each module does: - `http_2xx` succeeds when the URL returns a 2xx status code after following redirects. For HTTPS targets it also records the certificate expiry date. - `http_keyword` also requires the response body to match a regular expression, which catches error pages served with a 200 status. Change `Example Domain` to a string that only appears on a healthy page of your site. - `tcp_connect` succeeds when a TCP connection opens, useful for SSH, databases or mail ports. - `icmp` sends a ping. It needs the `CAP_NET_RAW` capability, which the systemd unit in the next step grants. - `dns_a` asks the target DNS server for the A record of `example.com` and expects a `NOERROR` answer. Replace the name with your own domain. Set the ownership and validate the file. The `--config.check` flag parses the configuration and exits: ```bash sudo chown -R root:blackbox /etc/blackbox_exporter sudo chmod 0750 /etc/blackbox_exporter sudo chmod 0640 /etc/blackbox_exporter/blackbox.yml blackbox_exporter --config.check --config.file=/etc/blackbox_exporter/blackbox.yml ``` A valid file produces a line containing `Config file is ok exiting...`. Any YAML or key error is reported with its line number. ## Step 3 - Running Blackbox Exporter with systemd Create a unit file so the exporter starts on boot and restarts if it fails: ```bash sudo nano /etc/systemd/system/blackbox_exporter.service ``` ```ini [Unit] Description=Prometheus Blackbox Exporter Wants=network-online.target After=network-online.target [Service] User=blackbox Group=blackbox ExecStart=/usr/local/bin/blackbox_exporter \ --config.file=/etc/blackbox_exporter/blackbox.yml \ --web.listen-address=127.0.0.1:9115 ExecReload=/bin/kill -HUP $MAINPID AmbientCapabilities=CAP_NET_RAW CapabilityBoundingSet=CAP_NET_RAW NoNewPrivileges=true Restart=on-failure [Install] WantedBy=multi-user.target ``` `AmbientCapabilities=CAP_NET_RAW` lets the unprivileged `blackbox` user open raw sockets for ICMP without running as root. The listen address keeps the exporter on localhost, because anyone who can reach `/probe` could use your server to probe arbitrary hosts. Reload systemd and start the service: ```bash sudo systemctl daemon-reload sudo systemctl enable --now blackbox_exporter sudo systemctl status blackbox_exporter ``` ```output ● blackbox_exporter.service - Prometheus Blackbox Exporter Loaded: loaded (/etc/systemd/system/blackbox_exporter.service; enabled; preset: enabled) Active: active (running) since ... ``` If the status is `failed`, read the log with `sudo journalctl -u blackbox_exporter -n 50`. ## Step 4 - Testing probes manually Before involving Prometheus, call the `/probe` endpoint directly. The `target` and `module` parameters are exactly what Prometheus will send: ```bash curl -s "http://127.0.0.1:9115/probe?target=https://example.com&module=http_2xx" | grep -E '^probe_(success|http_status_code|duration_seconds|ssl_earliest_cert_expiry) ' ``` ```output probe_duration_seconds 0.183412 probe_http_status_code 200 probe_ssl_earliest_cert_expiry 1.7704224e+09 probe_success 1 ``` `probe_success 1` means the probe passed. `probe_ssl_earliest_cert_expiry` is a Unix timestamp of the first certificate in the chain to expire. Test the other modules the same way: ```bash curl -s "http://127.0.0.1:9115/probe?target=example.com:443&module=tcp_connect" | grep '^probe_success' curl -s "http://127.0.0.1:9115/probe?target=1.1.1.1&module=icmp" | grep '^probe_success' curl -s "http://127.0.0.1:9115/probe?target=1.1.1.1:53&module=dns_a" | grep '^probe_success' ``` Each should print `probe_success 1`. When a probe fails, add `&debug=true` to the URL: the exporter returns a step-by-step log of the probe (DNS resolution, connection, TLS handshake, status code) instead of metrics, which usually shows the cause immediately. ## Step 5 - Scraping probes from Prometheus Prometheus does not scrape the targets directly. Each scrape job sends requests to Blackbox Exporter and uses relabeling to move the real target into the `target` URL parameter and the `instance` label. Open the Prometheus configuration: ```bash sudo nano /etc/prometheus/prometheus.yml ``` Add these jobs under the existing `scrape_configs:` key, keeping the indentation of the other jobs: ```yaml - job_name: blackbox_http metrics_path: /probe params: module: [http_2xx] static_configs: - targets: - https://example.com - https://your_domain relabel_configs: - source_labels: [__address__] target_label: __param_target - source_labels: [__param_target] target_label: instance - target_label: __address__ replacement: 127.0.0.1:9115 - job_name: blackbox_icmp metrics_path: /probe params: module: [icmp] static_configs: - targets: - 1.1.1.1 - your_server_ip relabel_configs: - source_labels: [__address__] target_label: __param_target - source_labels: [__param_target] target_label: instance - target_label: __address__ replacement: 127.0.0.1:9115 - job_name: blackbox_exporter static_configs: - targets: ['127.0.0.1:9115'] ``` The three relabel rules copy the listed target into `__param_target` (the `?target=` parameter), keep it as the `instance` label so graphs show the real endpoint, and finally point the scrape at the exporter. The last job scrapes the exporter's own metrics, such as its configuration reload status. Validate the configuration and restart Prometheus: ```bash promtool check config /etc/prometheus/prometheus.yml sudo systemctl restart prometheus ``` ```output Checking /etc/prometheus/prometheus.yml SUCCESS: /etc/prometheus/prometheus.yml is valid prometheus config file syntax ``` After one scrape interval, query the results from the Prometheus API: ```bash curl -s 'http://localhost:9090/api/v1/query' --data-urlencode 'query=probe_success{job="blackbox_http"}' ``` The response contains one result per target, with `"instance":"https://example.com"` and a value of `"1"` when the probe succeeds. You can also open the Prometheus web interface, go to **Status > Targets** and confirm that the `blackbox_http` and `blackbox_icmp` targets are `UP`. A target shows as `UP` even when the probe fails, because the scrape of the exporter itself worked: always alert on `probe_success`, not on `up`. ## Step 6 - Adding alerting rules Create a directory for rule files and a rules file for the probes: ```bash sudo mkdir -p /etc/prometheus/rules sudo nano /etc/prometheus/rules/blackbox.yml ``` ```yaml groups: - name: blackbox rules: - alert: EndpointDown expr: probe_success == 0 for: 2m labels: severity: critical annotations: summary: "{{ $labels.instance }} is not responding" description: "The {{ $labels.job }} probe for {{ $labels.instance }} has failed for more than 2 minutes." - alert: EndpointSlow expr: avg_over_time(probe_duration_seconds[5m]) > 2 for: 5m labels: severity: warning annotations: summary: "{{ $labels.instance }} is slow" description: "Average probe duration has been above 2 seconds for 5 minutes." - alert: TLSCertificateExpiringSoon expr: probe_ssl_earliest_cert_expiry - time() < 14 * 86400 for: 1h labels: severity: warning annotations: summary: "TLS certificate for {{ $labels.instance }} expires soon" description: "The certificate expires in {{ $value | humanizeDuration }}." ``` The `for` clause prevents a single failed probe from firing an alert. Fourteen days before expiry gives you time to react if automatic renewal (for example with Certbot) is broken. Tell Prometheus to load the file. In `/etc/prometheus/prometheus.yml`, set the top-level `rule_files` key (replace the commented example if your file has one): ```yaml rule_files: - /etc/prometheus/rules/*.yml ``` Check the rules and the configuration, then restart Prometheus: ```bash promtool check rules /etc/prometheus/rules/blackbox.yml promtool check config /etc/prometheus/prometheus.yml sudo systemctl restart prometheus ``` ```output Checking /etc/prometheus/rules/blackbox.yml SUCCESS: 3 rules found ``` The rules now appear under **Alerts** in the Prometheus web interface. To send notifications by email or chat, point Prometheus at an Alertmanager instance or create equivalent alert rules in Grafana. To test the `EndpointDown` alert, add a target that does not exist, such as `https://does-not-exist.invalid`, to the `blackbox_http` job and restart Prometheus. The alert goes to `Pending` on the next evaluation and to `Firing` two minutes later. ## Troubleshooting - **ICMP probes fail with `permission denied` or `socket: operation not permitted`**: the service is missing the raw socket capability. Confirm the `AmbientCapabilities=CAP_NET_RAW` line is in the unit, then run `sudo systemctl daemon-reload` and `sudo systemctl restart blackbox_exporter`. - **HTTP probes fail but `curl` from the server works**: add `&debug=true` to the probe URL. Common causes are an IPv6 address being tried first (keep `preferred_ip_protocol: ip4` if the host has no IPv6 route) and redirects to another host that fails. - **`context deadline exceeded`**: the module timeout is longer than the Prometheus `scrape_timeout` (10 seconds by default) or the target is slow. Keep the module timeout below the scrape timeout. - **Configuration changes are ignored**: after editing `blackbox.yml`, run `sudo systemctl reload blackbox_exporter`. The exporter re-reads the file on `SIGHUP` and logs an error if it is invalid. ## Conclusion Blackbox Exporter now probes your endpoints over HTTP, ICMP, TCP and DNS, Prometheus stores the results, and alert rules catch outages, slow responses and certificates close to expiry. Because the probes run from outside the application, they detect problems that internal metrics miss, such as DNS or TLS errors. As next steps, run a second Blackbox Exporter from a different location so you can tell a real outage from a network problem at one site, send the alerts through Alertmanager or Grafana alerting, and build a Grafana dashboard on `probe_success` and `probe_duration_seconds` for an availability overview.