--- title: "HAProxy Health Checks and Failover on Ubuntu 24.04" description: "Set up HTTP and TCP health checks in HAProxy on Ubuntu 24.04, tune inter, fall and rise, add backup servers and test automatic failover and recovery." category: "Proxy and Load Balancing" language: en source: https://cubepath.com/docs/proxy-and-load-balancing/haproxy-health-checks-and-failover updated: 2026-09-25 --- # HAProxy Health Checks and Failover on Ubuntu 24.04 HAProxy actively probes every backend server and removes it from rotation as soon as it stops answering correctly, then brings it back when it recovers. In this tutorial you will configure HTTP health checks against a dedicated `/health` endpoint on Ubuntu 24.04, tune how fast failures are detected, add a backup server, and watch failover happen through the logs, the stats page and the runtime API. ## Prerequisites To follow this tutorial you need: - A server running Ubuntu 24.04 LTS (for example a CubePath VPS). Debian 12 works the same way. - A non-root user with `sudo` privileges. - Python 3, which is preinstalled on Ubuntu, to run small test backends. Ubuntu 24.04 ships HAProxy 2.8 LTS, which supports every directive used here. ## Step 1 - Installing HAProxy and socat Install HAProxy and `socat`, which you will use to talk to the HAProxy runtime API: ```bash sudo apt update sudo apt install haproxy socat ``` Check the version and that the service is running: ```bash haproxy -v | head -n 1 systemctl is-active haproxy ``` ```output HAProxy version 2.8.5-1ubuntu3 2024/04/01 - https://haproxy.org/ active ``` ## Step 2 - Starting three test backends Create a web root for each backend. Each contains an `index.html` with its name and a `health` file that the health check will request: ```bash for n in 1 2 3; do sudo mkdir -p "/srv/web$n" echo "web$n" | sudo tee "/srv/web$n/index.html" > /dev/null echo "OK" | sudo tee "/srv/web$n/health" > /dev/null done ``` Start them as transient systemd services with Python's built-in web server. `web1` and `web2` are the primary servers, `web3` will be the backup: ```bash sudo systemd-run --unit=web1 python3 -m http.server 8081 --bind 127.0.0.1 --directory /srv/web1 sudo systemd-run --unit=web2 python3 -m http.server 8082 --bind 127.0.0.1 --directory /srv/web2 sudo systemd-run --unit=web3 python3 -m http.server 8083 --bind 127.0.0.1 --directory /srv/web3 ``` Verify that each one answers on `/health`: ```bash curl -s http://127.0.0.1:8081/health http://127.0.0.1:8082/health http://127.0.0.1:8083/health ``` ```output OK OK OK ``` In production, `/health` should be an endpoint in your application that checks what the app really needs (database connection, disk, queue) and returns `200` only when it can serve traffic. ## Step 3 - Configuring HTTP health checks Open the HAProxy configuration: ```bash sudo nano /etc/haproxy/haproxy.cfg ``` Keep the existing `global` and `defaults` sections. The Ubuntu package already defines `mode http`, logging to `/dev/log` and the runtime API socket at `/run/haproxy/admin.sock`. Append the following at the end of the file: ```haproxy frontend web_front bind :80 default_backend web_back backend web_back balance roundrobin option httpchk http-check send meth GET uri /health ver HTTP/1.1 hdr Host localhost http-check expect status 200 default-server inter 3s fall 3 rise 2 server web1 127.0.0.1:8081 check server web2 127.0.0.1:8082 check server web3 127.0.0.1:8083 check backup listen stats bind 127.0.0.1:8404 stats enable stats uri / stats refresh 5s ``` What each health check line does: - `option httpchk` switches the checks from a plain TCP connect to an HTTP request. - `http-check send` defines the request: method, path, HTTP version and a `Host` header, which many applications require. - `http-check expect status 200` marks the server as failed on any other status code. You can also match a range with `expect rstatus ^2` or the body with `expect string OK`. - `default-server inter 3s fall 3 rise 2` applies to every `server` line: check every 3 seconds, mark a server DOWN after 3 consecutive failures and UP again after 2 consecutive successes. - `check` on each `server` line enables checking for that server. Without it, HAProxy never checks the server and always considers it UP. - `backup` keeps `web3` out of rotation until no primary server is UP. With these values, a dead server is detected in about 9 seconds (3 checks x 3 seconds). Lower `inter` detects failures faster at the cost of more check traffic. `fastinter` and `downinter` let you check more often while a server is changing state or while it is down. Validate the file and reload HAProxy: ```bash sudo haproxy -c -f /etc/haproxy/haproxy.cfg sudo systemctl reload haproxy ``` ```output Configuration file is valid ``` Send a few requests. Only the primary servers answer: ```bash for i in $(seq 4); do curl -s http://localhost/; done ``` ```output web1 web2 web1 web2 ``` ## Step 4 - Checking server status Query the status column of each server through the runtime API. Field 1 is the backend, field 2 the server and field 18 the status: ```bash echo "show stat" | sudo socat stdio /run/haproxy/admin.sock | cut -d, -f1,2,18 | grep web_back ``` ```output web_back,web1,UP web_back,web2,UP web_back,web3,UP web_back,BACKEND,UP ``` For a visual view, open the stats page. It only listens on localhost, so reach it through an SSH tunnel from your computer: ```bash ssh -L 8404:127.0.0.1:8404 your_user@your_server_ip ``` Then browse to `http://localhost:8404/`. Each server shows its state, the result of the last check (for example `L7OK/200`) and how long it has been in that state. ## Step 5 - Testing failover Simulate an application failure on `web1` by removing its health file. The process is still running, but `/health` now returns `404`: ```bash sudo rm /srv/web1/health ``` Follow the HAProxy log. After about 9 seconds HAProxy marks the server DOWN: ```bash sudo tail -f /var/log/haproxy.log ``` ```output 2026-09-25T10:14:02.118+00:00 your_hostname haproxy[1520]: Server web_back/web1 is DOWN, reason: Layer7 wrong status, code: 404, info: "File not found", check duration: 1ms. 1 active and 1 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue. ``` Press `Ctrl+C` and confirm that all traffic now goes to `web2`: ```bash for i in $(seq 4); do curl -s http://localhost/; done ``` ```output web2 web2 web2 web2 ``` Now stop `web2` completely. HAProxy's check gets `Connection refused` and marks it DOWN with a Layer4 error: ```bash sudo systemctl stop web2 ``` After a few seconds the backup server takes over: ```bash for i in $(seq 2); do curl -s http://localhost/; done ``` ```output web3 web3 ``` Restore both servers: ```bash echo "OK" | sudo tee /srv/web1/health > /dev/null sudo systemd-run --unit=web2 python3 -m http.server 8082 --bind 127.0.0.1 --directory /srv/web2 ``` After two successful checks (about 6 seconds with `rise 2`), both are back and `web3` returns to standby: ```bash echo "show stat" | sudo socat stdio /run/haproxy/admin.sock | cut -d, -f1,2,18 | grep web_back ``` ```output web_back,web1,UP web_back,web2,UP web_back,web3,UP web_back,BACKEND,UP ``` ## Step 6 - Tuning failover behavior A few extra options make failover smoother. Add them to the `web_back` backend: ```haproxy option redispatch retries 2 default-server inter 3s fall 3 rise 2 on-marked-down shutdown-sessions slowstart 30s ``` - `option redispatch` lets HAProxy send a request to another server if the connection to the chosen one fails, even when a persistence cookie points to the dead server. - `retries 2` sets how many connection attempts HAProxy makes before giving up. - `on-marked-down shutdown-sessions` closes open connections to a server as soon as it is marked DOWN, so long-lived clients reconnect to a healthy server instead of hanging. - `slowstart 30s` ramps traffic to a recovered server up over 30 seconds instead of sending it a full share at once, which protects applications with cold caches. If you have several backup servers and want all of them to share the load when the primaries fail, add `option allbackups`. Without it, only the first backup receives traffic. Validate and reload after the change: ```bash sudo haproxy -c -f /etc/haproxy/haproxy.cfg && sudo systemctl reload haproxy ``` ## Step 7 - Health checking TCP services For non-HTTP services, run the backend in `mode tcp`. A plain `check` then verifies that a TCP connection can be established. Some protocols have dedicated checks, such as `option redis-check`, which sends a `PING` and expects `+PONG`: ```haproxy listen redis bind 127.0.0.1:6380 mode tcp option redis-check server redis1 10.0.0.21:6379 check inter 2s fall 3 rise 2 server redis2 10.0.0.22:6379 check inter 2s fall 3 rise 2 backup ``` Replace the addresses with your own Redis servers. A TCP connect check only proves that the port is open. When possible, prefer a protocol-level check or an HTTP health endpoint exposed by the service. ## Step 8 - Draining a server for maintenance Health checks handle unplanned failures. For planned work, drain the server through the runtime API: it stops receiving new connections but finishes the current ones. ```bash echo "set server web_back/web1 state drain" | sudo socat stdio /run/haproxy/admin.sock ``` When the maintenance is over, put it back into rotation: ```bash echo "set server web_back/web1 state ready" | sudo socat stdio /run/haproxy/admin.sock ``` Use `state maint` instead of `drain` to stop traffic and checks immediately. These changes are not saved to the configuration and are lost on a restart. ## Troubleshooting - **Server is DOWN with `Layer7 wrong status, code: 400`**: the application rejects the check request, usually because of a missing or wrong `Host` header. Adjust `hdr Host` in `http-check send`. - **Server is DOWN with `Layer4 connection problem`**: nothing listens on that address and port, or a firewall blocks HAProxy. Test from the HAProxy host with `curl -v http://backend_ip:port/health`. - **Server is always UP even though it is broken**: the `server` line is missing the `check` keyword, or the health endpoint returns `200` without testing anything. - **`/var/log/haproxy.log` is empty**: HAProxy logs through rsyslog on Ubuntu. Check `systemctl status rsyslog` and restart it with `sudo systemctl restart rsyslog`. ## Conclusion HAProxy now checks every backend over HTTP, removes failed servers in about 9 seconds, promotes a backup server when all primaries are down, and brings recovered servers back gradually. You also learned how to read server state from the runtime API and drain servers for maintenance. As next steps, point the checks at a real application health endpoint, enable TLS on the frontend, and run two HAProxy nodes with keepalived and a floating IP so the load balancer itself is not a single point of failure.