How to Set Up a GlusterFS Replicated Volume on Ubuntu 24.04

GlusterFS is a network filesystem that combines directories (called bricks) on several servers into a single volume that clients mount like a local disk. In this tutorial you will build a three-node GlusterFS cluster on Ubuntu 24.04, create a replicated volume that keeps a full copy of every file on each node, mount it on a client, and check that the data stays available when one node goes down.

Prerequisites

To follow this guide you need:

• Three servers running Ubuntu 24.04 LTS for the storage nodes, for example three CubePath VPS instances. This guide calls them gluster1, gluster2 and gluster3.
• One more Ubuntu 24.04 server that will mount the volume (the client). It can be an application server you already have.
• A non-root user with sudo privileges on every server.
• A second, empty disk on each storage node (this guide uses /dev/vdb). A brick can live on the root disk, but GlusterFS warns against it and a full brick would then fill your system disk.
• A private network between all four servers. GlusterFS traffic is not encrypted by default, so do not expose it on public interfaces.

The examples use these private addresses. Replace them with your own:

Understanding GlusterFS volume types

Before creating anything, pick the volume type. The most common ones are:

This guide uses a three-way replicated volume. It is the simplest layout that tolerates the loss of a node without split-brain problems, because three copies always give a majority.

Step 1 - Configuring hostname resolution

GlusterFS identifies peers and bricks by hostname, so every node and the client must resolve all storage node names to their private IPs. On all four servers, open /etc/hosts:

sudo nano /etc/hosts

Add these lines at the end:

10.0.0.11 gluster1
10.0.0.12 gluster2
10.0.0.13 gluster3

Check that the names resolve from each server:

ping -c 2 gluster2

PING gluster2 (10.0.0.12) 56(84) bytes of data.
64 bytes from gluster2 (10.0.0.12): icmp_seq=1 ttl=64 time=0.412 ms
64 bytes from gluster2 (10.0.0.12): icmp_seq=2 ttl=64 time=0.387 ms

If you use internal DNS instead of /etc/hosts, that works too, as long as the names are resolvable before you create the cluster.

Step 2 - Preparing the brick filesystem

Run this step on gluster1, gluster2 and gluster3. GlusterFS recommends XFS for bricks, formatted with 512-byte inodes so the extended attributes GlusterFS stores fit inside the inode.

First confirm which disk is empty. The target disk must have no partitions and no mount point:

lsblk -l -o NAME,SIZE,TYPE,MOUNTPOINTS

NAME    SIZE TYPE MOUNTPOINTS
vda      40G disk
vda1     39G part /
vda15   106M part /boot/efi
vdb     100G disk

Warning

The next command erases the disk. Double check the device name before running it.

Install the XFS tools and format the disk:

sudo apt update
sudo apt install xfsprogs
sudo mkfs.xfs -i size=512 /dev/vdb

Create the mount point and add the filesystem to /etc/fstab by UUID, so the mount survives device renaming:

sudo mkdir -p /data/glusterfs/gv0
echo "UUID=$(sudo blkid -s UUID -o value /dev/vdb) /data/glusterfs/gv0 xfs defaults 0 0" | sudo tee -a /etc/fstab
sudo systemctl daemon-reload
sudo mount -a

Create the brick directory inside the mount point. Using a subdirectory instead of the mount point itself means that if the disk ever fails to mount, GlusterFS finds no brick directory and refuses to start the brick, instead of silently writing to the root disk:

sudo mkdir -p /data/glusterfs/gv0/brick

Verify the mount:

df -h /data/glusterfs/gv0

Filesystem      Size  Used Avail Use% Mounted on
/dev/vdb        100G  746M  100G   1% /data/glusterfs/gv0

Step 3 - Installing the GlusterFS server

Ubuntu 24.04 ships GlusterFS 11 in its universe repository. On all three storage nodes, install the server package and start the management daemon:

sudo apt install glusterfs-server
sudo systemctl enable --now glusterd

Check that the daemon is running:

systemctl status glusterd --no-pager

● glusterd.service - GlusterFS, a clustered file-system server
     Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; preset: enabled)
     Active: active (running) since Thu 2026-09-24 10:12:40 UTC; 5s ago

Confirm the installed version on each node. All nodes should run the same version:

gluster --version | head -n 1

glusterfs 11.1

Step 4 - Opening the firewall on the private network

GlusterFS uses TCP port 24007 for the management daemon and one port per brick starting at 49152. Allow those ports only from your private subnet. On all three storage nodes:

sudo ufw allow from 10.0.0.0/24 to any port 24007:24008 proto tcp
sudo ufw allow from 10.0.0.0/24 to any port 49152:60999 proto tcp

If UFW is not active yet, allow SSH first so you do not lock yourself out, then enable it:

sudo ufw allow OpenSSH
sudo ufw enable

Verify the rules:

sudo ufw status

Status: active

To                         Action      From
--                         ------      ----
OpenSSH                    ALLOW       Anywhere
24007:24008/tcp            ALLOW       10.0.0.0/24
49152:60999/tcp            ALLOW       10.0.0.0/24

Step 5 - Creating the trusted storage pool

The storage nodes must join a trusted pool before they can share a volume. You only run this on gluster1; it adds the other two nodes:

sudo gluster peer probe gluster2
sudo gluster peer probe gluster3

peer probe: success
peer probe: success

Check the pool state:

sudo gluster peer status

Number of Peers: 2

Hostname: gluster2
Uuid: 3f7a1c52-9e1d-4b6a-8c1f-2d7b9a6e4c10
State: Peer in Cluster (Connected)

Hostname: gluster3
Uuid: 8b2e6d41-5a3c-4f7e-9d0b-1c6a8e2f7b93
State: Peer in Cluster (Connected)

Every peer must show Peer in Cluster (Connected). Running sudo gluster pool list on any node shows all three members, including localhost.

Step 6 - Creating and starting the replicated volume

Create a volume called gv0 with one brick on each node and three copies of every file. Run this on gluster1:

sudo gluster volume create gv0 replica 3 \
  gluster1:/data/glusterfs/gv0/brick \
  gluster2:/data/glusterfs/gv0/brick \
  gluster3:/data/glusterfs/gv0/brick

volume create: gv0: success: please start the volume to access data

Start the volume:

sudo gluster volume start gv0

By default any host that can reach the brick ports can mount the volume. Restrict access to your private subnet with the auth.allow option, which accepts wildcards and comma-separated addresses:

sudo gluster volume set gv0 auth.allow 10.0.0.*

Check the volume configuration:

sudo gluster volume info gv0

Volume Name: gv0
Type: Replicate
Status: Started
Number of Bricks: 1 x 3 = 3
Transport-type: tcp
Bricks:
Brick1: gluster1:/data/glusterfs/gv0/brick
Brick2: gluster2:/data/glusterfs/gv0/brick
Brick3: gluster3:/data/glusterfs/gv0/brick
Options Reconfigured:
auth.allow: 10.0.0.*
...

Then check that every brick process is online:

sudo gluster volume status gv0

The Online column must show Y for all three bricks and for the self-heal daemons.

Step 7 - Mounting the volume on the client

On the client, install the native FUSE client. It talks to all bricks directly, so the mount keeps working when one node is down:

sudo apt update
sudo apt install glusterfs-client

Create a mount point and mount the volume. The server you name here is only used to fetch the volume layout:

sudo mkdir -p /mnt/gv0
sudo mount -t glusterfs gluster1:/gv0 /mnt/gv0

Verify the mount:

df -h /mnt/gv0

Filesystem      Size  Used Avail Use% Mounted on
gluster1:/gv0   100G  1.8G   99G   2% /mnt/gv0

The size equals one brick, because each file is stored three times.

To mount the volume at boot, add it to /etc/fstab. The _netdev option waits for the network, and backup-volfile-servers lets the client fetch the layout from another node if gluster1 is down at boot time:

sudo nano /etc/fstab

gluster1:/gv0 /mnt/gv0 glusterfs defaults,_netdev,backup-volfile-servers=gluster2:gluster3 0 0

Test the entry without rebooting:

sudo umount /mnt/gv0
sudo systemctl daemon-reload
sudo mount -a
findmnt /mnt/gv0

TARGET   SOURCE        FSTYPE         OPTIONS
/mnt/gv0 gluster1:/gv0 fuse.glusterfs rw,relatime,user_id=0,group_id=0,default_permissions,allow_other,max_read=131072

Step 8 - Testing replication and node failure

Write some test files from the client:

for i in $(seq 1 20); do echo "file $i" | sudo tee "/mnt/gv0/test-$i.txt" > /dev/null; done
ls /mnt/gv0 | wc -l

20

On any storage node, look inside the brick. All 20 files are there, because each node holds a full copy:

ls /data/glusterfs/gv0/brick | wc -l

20

Important

Never write to a brick directory directly. Always go through a client mount, otherwise GlusterFS does not track the change.

Now simulate a failure. Shut down gluster3:

sudo poweroff

On the client, the volume keeps working. Write more files:

for i in $(seq 21 30); do echo "file $i" | sudo tee "/mnt/gv0/test-$i.txt" > /dev/null; done
ls /mnt/gv0 | wc -l

30

Start gluster3 again from your provider panel. When it is back, the self-heal daemon copies the 10 missing files to its brick. Check the pending heal entries from any node:

sudo gluster volume heal gv0 info

Brick gluster1:/data/glusterfs/gv0/brick
Status: Connected
Number of entries: 0

Brick gluster2:/data/glusterfs/gv0/brick
Status: Connected
Number of entries: 0

Brick gluster3:/data/glusterfs/gv0/brick
Status: Connected
Number of entries: 0

Right after the node returns you may see a few entries under gluster1 and gluster2. They drop to 0 once healing finishes, and ls /data/glusterfs/gv0/brick | wc -l on gluster3 then returns 30.

Step 9 - Expanding the volume

A replicated volume grows in sets of bricks equal to the replica count. To double the capacity of gv0, add one new brick on each of three nodes (new nodes or new disks on the existing ones), then rebalance so existing files spread across both sets:

sudo gluster volume add-brick gv0 \
  gluster1:/data/glusterfs/gv0b/brick \
  gluster2:/data/glusterfs/gv0b/brick \
  gluster3:/data/glusterfs/gv0b/brick
sudo gluster volume rebalance gv0 start

Follow the progress until every node reports completed:

sudo gluster volume rebalance gv0 status

The volume type changes to Distributed-Replicate with 2 x 3 = 6 bricks. Prepare each new brick exactly as in Step 2 before adding it.

Troubleshooting

peer probe: failed: Probe returned with Transport endpoint is not connected. The other node is not reachable on port 24007. Check that glusterd is running on it, that the hostname resolves to the private IP, and that the UFW rule from Step 4 exists on both sides.

volume create: gv0: failed: The brick ... is being created in the root partition. The brick path is not on the dedicated disk, usually because the disk did not mount. Run findmnt /data/glusterfs/gv0 and fix the mount instead of adding force.

The client mount hangs or fails. Read the client log, which is named after the mount point:

sudo tail -n 50 /var/log/glusterfs/mnt-gv0.log

Errors about brick ports mean the client cannot reach the 49152 and higher ports on some node. Errors about auth.allow mean the client IP is not in the allowed range.

Heal entries do not go down. Check for split-brain files, which need manual resolution:

sudo gluster volume heal gv0 info split-brain

Server logs are in /var/log/glusterfs/glusterd.log, and each brick has its own log in /var/log/glusterfs/bricks/.

Conclusion

You now have a three-node GlusterFS cluster serving a replicated volume that keeps working when a node fails, mounted on a client with automatic failover for the volume layout. From here you can move application data such as uploads or shared web content onto /mnt/gv0, enable TLS for the management and I/O paths if the traffic crosses untrusted networks, and monitor gluster volume status and gluster volume heal gv0 info from your monitoring system.