Pacemaker and Corosync are the standard open source stack for high availability on Linux. Corosync handles cluster membership and quorum: it knows which nodes are alive and whether enough of them agree. Pacemaker is the resource manager: it decides where services run and moves them when a node fails. In this tutorial you will build a three-node cluster on Ubuntu 24.04 with pcs, configure fencing, and run Nginx behind a floating IP that moves to a healthy node automatically.

Prerequisites

  • Three servers running Ubuntu 24.04 LTS, called node1, node2 and node3 in this guide. Three nodes let the cluster keep quorum when one fails; two-node clusters work but need special handling.
  • A non-root user with sudo privileges on each node.
  • A private network shared by all nodes at layer 2. This guide uses 10.0.0.11, 10.0.0.12 and 10.0.0.13.
  • A free address in that subnet for the floating IP, here 10.0.0.100.
  • A working fencing method. On bare metal servers this is usually the BMC (IPMI) of each node; this guide uses fence_ipmilan.

Step 1 - Preparing the nodes

Run this step on all three nodes.

Each node must resolve the others by name over the private network. Add them to /etc/hosts:

sudo nano /etc/hosts

10.0.0.11 node1
10.0.0.12 node2
10.0.0.13 node3

Make sure the hostname of each server matches its entry (hostnamectl shows it; sudo hostnamectl set-hostname node1 changes it).

Cluster logs and fencing decisions are much easier to follow when clocks agree. Ubuntu 24.04 synchronizes time with systemd-timesyncd by default; confirm the clock is synchronized:

timedatectl

System clock synchronized: yes
              NTP service: active

Open the cluster ports to the private network only. pcsd uses TCP 2224 and Corosync uses UDP 5405 by default:

sudo ufw allow from 10.0.0.0/24 to any port 2224 proto tcp
sudo ufw allow from 10.0.0.0/24 to any port 5405 proto udp

Step 2 - Installing Pacemaker, Corosync and pcs

On all three nodes, install the cluster stack, the pcs management tool, the base resource agents (which include IPaddr2) and the base fence agents (which include fence_ipmilan):

sudo apt update
sudo apt install pacemaker corosync pcs resource-agents-base fence-agents-base

The pcs package creates the hacluster user that pcsd uses to authenticate nodes to each other. Give it the same strong password on every node:

sudo passwd hacluster

Ubuntu's pacemaker package starts with a placeholder single-node configuration. Remove it so pcs can create the real cluster:

sudo pcs cluster destroy

Make sure pcsd is running and enabled:

sudo systemctl enable --now pcsd
sudo systemctl status pcsd --no-pager

● pcsd.service - PCS GUI and remote configuration interface
     Loaded: loaded (/usr/lib/systemd/system/pcsd.service; enabled; preset: enabled)
     Active: active (running)

Step 3 - Creating the cluster

Run the rest of the pcs commands on node1 only; pcs pushes the configuration to the other nodes.

Authenticate the nodes to each other. Enter the hacluster password when prompted:

sudo pcs host auth node1 addr=10.0.0.11 node2 addr=10.0.0.12 node3 addr=10.0.0.13 -u hacluster

node1: Authorized
node2: Authorized
node3: Authorized

Create the cluster, start it on all nodes and enable it at boot:

sudo pcs cluster setup hacluster01 node1 addr=10.0.0.11 node2 addr=10.0.0.12 node3 addr=10.0.0.13
sudo pcs cluster start --all
sudo pcs cluster enable --all

pcs writes /etc/corosync/corosync.conf on every node with the knet transport and the node addresses you gave.

Verify membership and quorum:

sudo pcs status

Cluster name: hacluster01
Status of pacemakerd: 'Pacemaker is running'
Cluster Summary:
  * Stack: corosync (Pacemaker is running)
  * Current DC: node1 (version 2.1.6-6fdc9deea29) - partition with quorum
  * 3 nodes configured
  * 0 resource instances configured

Node List:
  * Online: [ node1 node2 node3 ]

You will also see a warning that no STONITH resources are defined. That is the next step.

sudo corosync-quorumtool

Votequorum information
----------------------
Expected votes:   3
Highest expected: 3
Total votes:      3
Quorum:           2
Flags:            Quorate

Step 4 - Configuring fencing (STONITH)

Fencing (STONITH, "shoot the other node in the head") is what makes a cluster safe. When a node stops responding, the cluster cannot know whether it crashed or is still running the service while cut off from the network. Before starting the service elsewhere, Pacemaker powers the lost node off through an out-of-band channel, so two nodes never run the same resource and never write to the same data at once.

Create one fence device per node, pointing at that node's BMC. Replace the BMC addresses, user and password with your own:

sudo pcs stonith create fence-node1 fence_ipmilan ip=10.0.1.11 username=your_bmc_user password=your_bmc_password lanplus=1 pcmk_host_list=node1
sudo pcs stonith create fence-node2 fence_ipmilan ip=10.0.1.12 username=your_bmc_user password=your_bmc_password lanplus=1 pcmk_host_list=node2
sudo pcs stonith create fence-node3 fence_ipmilan ip=10.0.1.13 username=your_bmc_user password=your_bmc_password lanplus=1 pcmk_host_list=node3

Before trusting a fence device, check that each node can reach the BMCs and read the power status:

sudo fence_ipmilan --ip=10.0.1.12 --username=your_bmc_user --password=your_bmc_password --lanplus --action=status

Status: ON

Keep a node from fencing itself by preferring other nodes to run each device:

sudo pcs constraint location fence-node1 avoids node1
sudo pcs constraint location fence-node2 avoids node2
sudo pcs constraint location fence-node3 avoids node3

Make sure fencing is enabled (it is by default) and test it on a node that is not running anything important yet:

sudo pcs property set stonith-enabled=true
sudo pcs stonith fence node3

node3 should power cycle, come back, and rejoin the cluster. Check with sudo pcs status.

Step 5 - Adding a floating IP and Nginx

Install Nginx on all three nodes and disable its systemd unit, because Pacemaker, not systemd, decides where it runs:

sudo apt install nginx
sudo systemctl disable --now nginx

So you can see which node answers, put the hostname in each node's default page:

hostname | sudo tee /var/www/html/index.html

Back on node1, set a default resource stickiness. It makes resources stay on their current node after a failed node comes back, instead of moving again and causing a second interruption:

sudo pcs resource defaults update resource-stickiness=100

Create the floating IP with the IPaddr2 agent and Nginx with the systemd class, both in one group. Resources in a group always run on the same node and start in order, so the IP comes up before Nginx:

sudo pcs resource create web-vip ocf:heartbeat:IPaddr2 ip=10.0.0.100 cidr_netmask=24 op monitor interval=10s --group web
sudo pcs resource create web-nginx systemd:nginx op monitor interval=15s --group web

Check where the group is running:

sudo pcs status resources

  * Resource Group: web:
    * web-vip	(ocf:heartbeat:IPaddr2):	 Started node1
    * web-nginx	(systemd:nginx):	 Started node1

From any machine on the private network, request the floating IP:

curl http://10.0.0.100

node1

Step 6 - Testing failover

Put the active node in standby. Pacemaker stops the resources there and starts them on another node:

sudo pcs node standby node1
sudo pcs status resources

  * Resource Group: web:
    * web-vip	(ocf:heartbeat:IPaddr2):	 Started node2
    * web-nginx	(systemd:nginx):	 Started node2

curl http://10.0.0.100 now returns node2. Bring node1 back:

sudo pcs node unstandby node1

Thanks to the stickiness you set, the group stays on node2.

Next, test a service failure. On the node running the group, stop Nginx behind Pacemaker's back:

sudo systemctl stop nginx

Within the monitor interval Pacemaker detects the failure and restarts Nginx. The failure is recorded:

sudo pcs resource failcount show web-nginx

By default a resource moves to another node only after it fails INFINITY times on a node, meaning it is restarted in place. To move it after three failures, set migration-threshold:

sudo pcs resource meta web-nginx migration-threshold=3

Finally, test a real node failure by powering off the active node (or pulling its network cable). The remaining two nodes keep quorum, fence the lost node and start the group on one of them.

Troubleshooting

Resources show as Stopped and pcs status warns about STONITH. Pacemaker refuses to start resources until fencing is configured. Complete Step 4.

A resource is stuck in a failed state. Check the reason and clear the failure history once the cause is fixed:

sudo pcs status --full
sudo journalctl -u pacemaker --since "15 min ago"
sudo pcs resource cleanup web-nginx

Nodes do not see each other. Check the Corosync link status on each node and make sure UDP 5405 is allowed between them:

sudo corosync-cfgtool -s

The output lists each link and should report the other nodes as connected.

The cluster lost quorum. With three nodes, the cluster stops resources when two nodes are unreachable, because it cannot rule out that they are running them. Bring the nodes back rather than forcing quorum; overriding quorum is exactly how split brain happens.

The floating IP moves but is not reachable. The network is probably filtering the gratuitous ARP or the address itself. Verify with ip -brief address on the active node that the IP is assigned, then check the network configuration with your provider.

Conclusion

You now have a three-node Pacemaker and Corosync cluster with fencing, a floating IP and Nginx that fails over when a node or the service fails. The same pattern applies to other services: add them as resources to a group, or use constraints for more complex layouts. As next steps, put HAProxy instead of Nginx behind the floating IP to load balance a pool of backends, add a shared or replicated storage resource such as DRBD for stateful services, and set up alerts on cluster events with Pacemaker's alert agents.