MariaDB Galera Cluster is a synchronous multi-master setup: every node accepts reads and writes, and a transaction is only committed once all nodes have certified it, so no data is lost when a node fails. In this tutorial you will build a three-node Galera cluster with the MariaDB 10.11 packages from Ubuntu 24.04, bootstrap it, check that writes replicate between nodes, and learn how to restart the cluster safely after all nodes go down.
Prerequisites
To follow this tutorial you need:
- Three servers running Ubuntu 24.04 LTS, such as three CubePath VPS, each with a non-root user that has
sudoprivileges and at least 2 GB of RAM. - A private network between the three servers. Galera traffic is not encrypted by default, so it should not travel over the public internet.
- The same MariaDB version on all nodes. This guide uses the Ubuntu packages on every node.
Always use an odd number of nodes, three at minimum. Galera needs a majority (quorum) to keep accepting writes: with three nodes, the cluster survives the loss of one; with two, losing either one stops the other.
The guide uses these names and private addresses. Replace them with yours:
| Node | Hostname | Private IP |
|---|---|---|
| 1 | node1 | 10.0.0.11 |
| 2 | node2 | 10.0.0.12 |
| 3 | node3 | 10.0.0.13 |
Step 1 - Installing MariaDB and Galera on all nodes
Run the commands in this step on all three nodes.
Install the MariaDB server, the Galera 4 replication library and mariadb-backup, which Galera will use to copy data to joining nodes:
sudo apt update
sudo apt install mariadb-server galera-4 mariadb-backup
The package starts MariaDB as a standalone server. Check the version:
mariadb --version
mariadb Ver 15.1 Distrib 10.11.8-MariaDB, for debian-linux-gnu (x86_64) using EditLine wrapper
Stop the service. A node must not be running when you change its cluster configuration, and the cluster has to be started in a specific order:
sudo systemctl stop mariadb
Step 2 - Opening the Galera ports
Galera uses four ports. Allow them only from the private network of the cluster, here 10.0.0.0/24, on all three nodes:
| Port | Protocol | Purpose |
|---|---|---|
| 3306 | TCP | MariaDB client connections |
| 4567 | TCP and UDP | Galera replication traffic |
| 4568 | TCP | Incremental State Transfer (IST) |
| 4444 | TCP | State Snapshot Transfer (SST) |
sudo ufw allow from 10.0.0.0/24 to any port 3306,4444,4567,4568 proto tcp
sudo ufw allow from 10.0.0.0/24 to any port 4567 proto udp
Check the rules:
sudo ufw status
To Action From
-- ---- ----
22/tcp ALLOW Anywhere
3306,4444,4567,4568/tcp ALLOW 10.0.0.0/24
4567/udp ALLOW 10.0.0.0/24
If UFW is not enabled, allow SSH first with sudo ufw allow OpenSSH and then run sudo ufw enable.
Step 3 - Configuring Galera on each node
Ubuntu's MariaDB package reads configuration files from /etc/mysql/mariadb.conf.d/ in alphabetical order, and it ships 60-galera.cnf as a commented template for exactly this purpose. Open it on node1:
sudo nano /etc/mysql/mariadb.conf.d/60-galera.cnf
Replace its contents with:
[mysqld]
# Listen on all interfaces so the other nodes and your apps can connect.
# UFW restricts port 3306 to the private network.
bind-address = 0.0.0.0
# Galera requirements
binlog_format = ROW
default_storage_engine = InnoDB
innodb_autoinc_lock_mode = 2
[galera]
wsrep_on = ON
wsrep_provider = /usr/lib/galera/libgalera_smm.so
# Identical on every node
wsrep_cluster_name = "galera_cluster"
wsrep_cluster_address = "gcomm://10.0.0.11,10.0.0.12,10.0.0.13"
# Unique on every node
wsrep_node_name = "node1"
wsrep_node_address = "10.0.0.11"
# How a new node receives a full copy of the data
wsrep_sst_method = mariabackup
wsrep_sst_auth = "sst_user:your_strong_password"
# Apply replicated transactions with several threads
wsrep_applier_threads = 4
What the key settings do:
binlog_format = ROWandinnodb_autoinc_lock_mode = 2are required by Galera: it replicates row changes, and interleaved auto-increment locking prevents deadlocks when several nodes insert at once.wsrep_cluster_addresslists all members. A starting node tries each address to find a running member.wsrep_sst_method = mariabackupcopies data to a joining node with a hot backup, so the donor node keeps accepting writes during the transfer. The olderrsyncmethod blocks the donor.wsrep_sst_authis the database account that mariadb-backup uses on the donor node. You will create it in Step 4. Replaceyour_strong_passwordwith a strong, unique password.wsrep_applier_threadssets how many threads apply replicated writes. A value around the number of CPU cores is a good start.
Because this file is read after 50-server.cnf, its bind-address overrides the default of 127.0.0.1.
Repeat on node2 and node3 with the same content, changing only the two node-specific lines:
# node2
wsrep_node_name = "node2"
wsrep_node_address = "10.0.0.12"
# node3
wsrep_node_name = "node3"
wsrep_node_address = "10.0.0.13"
Step 4 - Bootstrapping the cluster on node1
The first node of a new cluster cannot join anything, so it has to be started in bootstrap mode, which creates a new cluster with itself as the only member. Ubuntu provides the galera_new_cluster script for this. Run it on node1 only:
sudo galera_new_cluster
Check the cluster size and status:
sudo mariadb -e "SHOW STATUS WHERE Variable_name IN ('wsrep_cluster_size','wsrep_cluster_status','wsrep_local_state_comment','wsrep_ready')"
+---------------------------+---------+
| Variable_name | Value |
+---------------------------+---------+
| wsrep_cluster_size | 1 |
| wsrep_cluster_status | Primary |
| wsrep_local_state_comment | Synced |
| wsrep_ready | ON |
+---------------------------+---------+
Now create the SST user referenced in wsrep_sst_auth. It only needs to connect locally on the donor, with the privileges mariadb-backup requires. Use the same password you put in the configuration file:
sudo mariadb
CREATE USER 'sst_user'@'localhost' IDENTIFIED BY 'your_strong_password';
GRANT RELOAD, PROCESS, LOCK TABLES, BINLOG MONITOR ON *.* TO 'sst_user'@'localhost';
EXIT;
The user is stored in the system tables, so it is copied to the other nodes when they join.
Step 5 - Joining node2 and node3
On node2, start MariaDB normally. It reads wsrep_cluster_address, contacts node1 and receives a full copy of the data through SST before it starts accepting connections:
sudo systemctl start mariadb
The first start takes longer than usual because of the transfer. Check the cluster size from any node:
sudo mariadb -e "SHOW STATUS LIKE 'wsrep_cluster_size'"
+--------------------+-------+
| Variable_name | Value |
+--------------------+-------+
| wsrep_cluster_size | 2 |
+--------------------+-------+
Then start MariaDB on node3 the same way:
sudo systemctl start mariadb
The cluster size is now 3. If a node fails to start, read its log while it tries to join:
sudo journalctl -u mariadb -n 50 --no-pager
Node1 is still running in bootstrap mode, which is fine. Bootstrap only matters at startup: if you restart node1 now, it starts normally and rejoins through the other two nodes.
Step 6 - Verifying synchronous replication
Create a database and a table on node1:
sudo mariadb -e "CREATE DATABASE galera_test; CREATE TABLE galera_test.items (id INT AUTO_INCREMENT PRIMARY KEY, name VARCHAR(50)); INSERT INTO galera_test.items (name) VALUES ('from node1');"
Insert a row on node2:
sudo mariadb -e "INSERT INTO galera_test.items (name) VALUES ('from node2');"
Read the table on node3:
sudo mariadb -e "SELECT * FROM galera_test.items;"
+----+------------+
| id | name |
+----+------------+
| 1 | from node1 |
| 3 | from node2 |
+----+------------+
Both rows are there. The IDs may not be consecutive because Galera sets auto_increment_increment to the cluster size and gives each node a different offset, so nodes never generate the same ID. Do not rely on IDs being sequential.
Remove the test database:
sudo mariadb -e "DROP DATABASE galera_test;"
Only InnoDB tables are replicated. Tables using other engines, such as MyISAM, exist only on the node where they were written.
Step 7 - Monitoring the cluster
These status variables tell you whether a node is healthy:
| Variable | Healthy value | Meaning |
|---|---|---|
wsrep_cluster_size | 3 | Number of nodes in the cluster |
wsrep_cluster_status | Primary | The node is part of the majority and accepts writes |
wsrep_local_state_comment | Synced | The node is up to date |
wsrep_ready | ON | The node accepts queries |
wsrep_flow_control_paused | close to 0 | Fraction of time replication was paused because a node could not keep up |
Query them all at once:
sudo mariadb -e "SHOW STATUS WHERE Variable_name IN ('wsrep_cluster_size','wsrep_cluster_status','wsrep_local_state_comment','wsrep_ready','wsrep_flow_control_paused')"
A wsrep_cluster_status of non-Primary means the node lost contact with the majority. It refuses queries to prevent a split brain, where two halves of the cluster accept different writes. It rejoins automatically once the network is back.
A value of wsrep_flow_control_paused that stays above 0.1 means one node is slowing down the whole cluster, usually because of slow disks or too few wsrep_applier_threads.
Step 8 - Restarting and recovering the cluster
Rolling restarts
For configuration changes or package updates, restart one node at a time and wait until it is Synced before moving to the next:
sudo systemctl restart mariadb
sudo mariadb -e "SHOW STATUS LIKE 'wsrep_local_state_comment'"
The restarted node catches up with a fast incremental transfer (IST) if the changes it missed are still in the donor's cache, or with a full SST otherwise. The cluster stays available the whole time.
Starting after all nodes were shut down
If all nodes are stopped, a plain systemctl start on any node waits forever for a cluster to join. You have to bootstrap again, and it must be from the node with the most recent data. Galera records this in /var/lib/mysql/grastate.dat. Check it on each node:
sudo cat /var/lib/mysql/grastate.dat
# GALERA saved state
version: 2.1
uuid: 8c6f5d2e-9a3b-11ef-9d3b-7a1c2f3e4d5a
seqno: 1843
safe_to_bootstrap: 1
After a clean shutdown, the last node that stopped has safe_to_bootstrap: 1. Run sudo galera_new_cluster on that node, then sudo systemctl start mariadb on the other two.
After a crash (power loss, for example), every node may show seqno: -1. Recover the last committed position on each node with:
sudo -u mysql galera_recovery
--wsrep_start_position=8c6f5d2e-9a3b-11ef-9d3b-7a1c2f3e4d5a:1843
The number after the colon is the node's last transaction. Pick the node with the highest value, edit its grastate.dat and set safe_to_bootstrap: 1:
sudo nano /var/lib/mysql/grastate.dat
Then bootstrap that node with sudo galera_new_cluster and start the others normally. Bootstrapping a node that does not have the latest data discards the transactions that only existed on the other nodes, so always compare the values first.
Troubleshooting
galera_new_cluster fails with "It may not be safe to bootstrap the cluster from this node". The node was not the last to leave. Check grastate.dat on every node as described in Step 8 and bootstrap from the node with the highest sequence number.
A joining node fails with an SST error. Check the log on both the joiner and the donor with sudo journalctl -u mariadb. The usual causes are a wrong password in wsrep_sst_auth, the sst_user account missing on the donor, mariadb-backup not installed on one of the nodes, or port 4444 blocked between nodes.
The node starts but wsrep_cluster_size stays at 1. The node created its own cluster instead of joining. Make sure you did not run galera_new_cluster on it, that wsrep_cluster_address lists the correct IPs, and that port 4567 is open in both TCP and UDP.
Writes fail with "Deadlock found when trying to get lock" under concurrent load. With multi-master writes, two nodes can modify the same row at the same time and one transaction loses the certification. Retry the transaction in your application, or send all writes to a single node and use the others for reads.
Conclusion
You now have a three-node MariaDB Galera Cluster on Ubuntu 24.04 that replicates every write synchronously, tolerates the loss of one node, and can be restarted safely even after a full outage. To use it from applications, put a load balancer in front of the nodes, such as HAProxy or MariaDB MaxScale, so clients always reach a healthy member. Also schedule regular backups with mariadb-backup --backup --galera-info on one node, because replication protects against hardware failure but not against a DROP TABLE.
