--- title: "How to Set Up a GlusterFS Replicated Volume on Ubuntu 24.04" description: "Build a three-node GlusterFS cluster on Ubuntu 24.04, create a replicated volume, mount it on a client and confirm that data survives the loss of one node." category: "Storage and Backup Advanced" language: en source: https://cubepath.com/docs/storage-and-backup-advanced/glusterfs-distributed-storage-configuration updated: 2026-09-25 --- # How to Set Up a GlusterFS Replicated Volume on Ubuntu 24.04 GlusterFS is a network filesystem that combines directories (called bricks) on several servers into a single volume that clients mount like a local disk. In this tutorial you will build a three-node GlusterFS cluster on Ubuntu 24.04, create a replicated volume that keeps a full copy of every file on each node, mount it on a client, and check that the data stays available when one node goes down. ## Prerequisites To follow this guide you need: - Three servers running Ubuntu 24.04 LTS for the storage nodes, for example three CubePath VPS instances. This guide calls them `gluster1`, `gluster2` and `gluster3`. - One more Ubuntu 24.04 server that will mount the volume (the client). It can be an application server you already have. - A non-root user with `sudo` privileges on every server. - A second, empty disk on each storage node (this guide uses `/dev/vdb`). A brick can live on the root disk, but GlusterFS warns against it and a full brick would then fill your system disk. - A private network between all four servers. GlusterFS traffic is not encrypted by default, so do not expose it on public interfaces. The examples use these private addresses. Replace them with your own: | Host | Private IP | |------|------------| | gluster1 | 10.0.0.11 | | gluster2 | 10.0.0.12 | | gluster3 | 10.0.0.13 | | client | 10.0.0.20 | ## Understanding GlusterFS volume types Before creating anything, pick the volume type. The most common ones are: | Type | How data is stored | Survives node loss | Usable capacity | |------|--------------------|--------------------|-----------------| | Distributed | Each file lives on one brick | No | Sum of all bricks | | Replicated (`replica 3`) | Every file on all three bricks | Yes, one or two nodes | One brick | | Arbiter (`replica 3 arbiter 1`) | Two full copies plus metadata on the third brick | Yes, one node | One brick, third node needs little space | | Dispersed (`disperse 3 redundancy 1`) | Erasure coded across bricks | Yes, one node | Two bricks | This guide uses a three-way replicated volume. It is the simplest layout that tolerates the loss of a node without split-brain problems, because three copies always give a majority. ## Step 1 - Configuring hostname resolution GlusterFS identifies peers and bricks by hostname, so every node and the client must resolve all storage node names to their private IPs. On **all four servers**, open `/etc/hosts`: ```bash sudo nano /etc/hosts ``` Add these lines at the end: ```text 10.0.0.11 gluster1 10.0.0.12 gluster2 10.0.0.13 gluster3 ``` Check that the names resolve from each server: ```bash ping -c 2 gluster2 ``` ```output PING gluster2 (10.0.0.12) 56(84) bytes of data. 64 bytes from gluster2 (10.0.0.12): icmp_seq=1 ttl=64 time=0.412 ms 64 bytes from gluster2 (10.0.0.12): icmp_seq=2 ttl=64 time=0.387 ms ``` If you use internal DNS instead of `/etc/hosts`, that works too, as long as the names are resolvable before you create the cluster. ## Step 2 - Preparing the brick filesystem Run this step on **gluster1, gluster2 and gluster3**. GlusterFS recommends XFS for bricks, formatted with 512-byte inodes so the extended attributes GlusterFS stores fit inside the inode. First confirm which disk is empty. The target disk must have no partitions and no mount point: ```bash lsblk -l -o NAME,SIZE,TYPE,MOUNTPOINTS ``` ```output NAME SIZE TYPE MOUNTPOINTS vda 40G disk vda1 39G part / vda15 106M part /boot/efi vdb 100G disk ``` > Warning: The next command erases the disk. Double check the device name before running it. Install the XFS tools and format the disk: ```bash sudo apt update sudo apt install xfsprogs sudo mkfs.xfs -i size=512 /dev/vdb ``` Create the mount point and add the filesystem to `/etc/fstab` by UUID, so the mount survives device renaming: ```bash sudo mkdir -p /data/glusterfs/gv0 echo "UUID=$(sudo blkid -s UUID -o value /dev/vdb) /data/glusterfs/gv0 xfs defaults 0 0" | sudo tee -a /etc/fstab sudo systemctl daemon-reload sudo mount -a ``` Create the brick directory inside the mount point. Using a subdirectory instead of the mount point itself means that if the disk ever fails to mount, GlusterFS finds no brick directory and refuses to start the brick, instead of silently writing to the root disk: ```bash sudo mkdir -p /data/glusterfs/gv0/brick ``` Verify the mount: ```bash df -h /data/glusterfs/gv0 ``` ```output Filesystem Size Used Avail Use% Mounted on /dev/vdb 100G 746M 100G 1% /data/glusterfs/gv0 ``` ## Step 3 - Installing the GlusterFS server Ubuntu 24.04 ships GlusterFS 11 in its universe repository. On **all three storage nodes**, install the server package and start the management daemon: ```bash sudo apt install glusterfs-server sudo systemctl enable --now glusterd ``` Check that the daemon is running: ```bash systemctl status glusterd --no-pager ``` ```output ● glusterd.service - GlusterFS, a clustered file-system server Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; preset: enabled) Active: active (running) since Thu 2026-09-24 10:12:40 UTC; 5s ago ``` Confirm the installed version on each node. All nodes should run the same version: ```bash gluster --version | head -n 1 ``` ```output glusterfs 11.1 ``` ## Step 4 - Opening the firewall on the private network GlusterFS uses TCP port 24007 for the management daemon and one port per brick starting at 49152. Allow those ports only from your private subnet. On **all three storage nodes**: ```bash sudo ufw allow from 10.0.0.0/24 to any port 24007:24008 proto tcp sudo ufw allow from 10.0.0.0/24 to any port 49152:60999 proto tcp ``` If UFW is not active yet, allow SSH first so you do not lock yourself out, then enable it: ```bash sudo ufw allow OpenSSH sudo ufw enable ``` Verify the rules: ```bash sudo ufw status ``` ```output Status: active To Action From -- ------ ---- OpenSSH ALLOW Anywhere 24007:24008/tcp ALLOW 10.0.0.0/24 49152:60999/tcp ALLOW 10.0.0.0/24 ``` ## Step 5 - Creating the trusted storage pool The storage nodes must join a trusted pool before they can share a volume. You only run this on **gluster1**; it adds the other two nodes: ```bash sudo gluster peer probe gluster2 sudo gluster peer probe gluster3 ``` ```output peer probe: success peer probe: success ``` Check the pool state: ```bash sudo gluster peer status ``` ```output Number of Peers: 2 Hostname: gluster2 Uuid: 3f7a1c52-9e1d-4b6a-8c1f-2d7b9a6e4c10 State: Peer in Cluster (Connected) Hostname: gluster3 Uuid: 8b2e6d41-5a3c-4f7e-9d0b-1c6a8e2f7b93 State: Peer in Cluster (Connected) ``` Every peer must show `Peer in Cluster (Connected)`. Running `sudo gluster pool list` on any node shows all three members, including `localhost`. ## Step 6 - Creating and starting the replicated volume Create a volume called `gv0` with one brick on each node and three copies of every file. Run this on **gluster1**: ```bash sudo gluster volume create gv0 replica 3 \ gluster1:/data/glusterfs/gv0/brick \ gluster2:/data/glusterfs/gv0/brick \ gluster3:/data/glusterfs/gv0/brick ``` ```output volume create: gv0: success: please start the volume to access data ``` Start the volume: ```bash sudo gluster volume start gv0 ``` By default any host that can reach the brick ports can mount the volume. Restrict access to your private subnet with the `auth.allow` option, which accepts wildcards and comma-separated addresses: ```bash sudo gluster volume set gv0 auth.allow 10.0.0.* ``` Check the volume configuration: ```bash sudo gluster volume info gv0 ``` ```output Volume Name: gv0 Type: Replicate Status: Started Number of Bricks: 1 x 3 = 3 Transport-type: tcp Bricks: Brick1: gluster1:/data/glusterfs/gv0/brick Brick2: gluster2:/data/glusterfs/gv0/brick Brick3: gluster3:/data/glusterfs/gv0/brick Options Reconfigured: auth.allow: 10.0.0.* ... ``` Then check that every brick process is online: ```bash sudo gluster volume status gv0 ``` The `Online` column must show `Y` for all three bricks and for the self-heal daemons. ## Step 7 - Mounting the volume on the client On the **client**, install the native FUSE client. It talks to all bricks directly, so the mount keeps working when one node is down: ```bash sudo apt update sudo apt install glusterfs-client ``` Create a mount point and mount the volume. The server you name here is only used to fetch the volume layout: ```bash sudo mkdir -p /mnt/gv0 sudo mount -t glusterfs gluster1:/gv0 /mnt/gv0 ``` Verify the mount: ```bash df -h /mnt/gv0 ``` ```output Filesystem Size Used Avail Use% Mounted on gluster1:/gv0 100G 1.8G 99G 2% /mnt/gv0 ``` The size equals one brick, because each file is stored three times. To mount the volume at boot, add it to `/etc/fstab`. The `_netdev` option waits for the network, and `backup-volfile-servers` lets the client fetch the layout from another node if `gluster1` is down at boot time: ```bash sudo nano /etc/fstab ``` ```text gluster1:/gv0 /mnt/gv0 glusterfs defaults,_netdev,backup-volfile-servers=gluster2:gluster3 0 0 ``` Test the entry without rebooting: ```bash sudo umount /mnt/gv0 sudo systemctl daemon-reload sudo mount -a findmnt /mnt/gv0 ``` ```output TARGET SOURCE FSTYPE OPTIONS /mnt/gv0 gluster1:/gv0 fuse.glusterfs rw,relatime,user_id=0,group_id=0,default_permissions,allow_other,max_read=131072 ``` ## Step 8 - Testing replication and node failure Write some test files from the client: ```bash for i in $(seq 1 20); do echo "file $i" | sudo tee "/mnt/gv0/test-$i.txt" > /dev/null; done ls /mnt/gv0 | wc -l ``` ```output 20 ``` On any storage node, look inside the brick. All 20 files are there, because each node holds a full copy: ```bash ls /data/glusterfs/gv0/brick | wc -l ``` ```output 20 ``` > Important: Never write to a brick directory directly. Always go through a client mount, otherwise GlusterFS does not track the change. Now simulate a failure. Shut down **gluster3**: ```bash sudo poweroff ``` On the client, the volume keeps working. Write more files: ```bash for i in $(seq 21 30); do echo "file $i" | sudo tee "/mnt/gv0/test-$i.txt" > /dev/null; done ls /mnt/gv0 | wc -l ``` ```output 30 ``` Start gluster3 again from your provider panel. When it is back, the self-heal daemon copies the 10 missing files to its brick. Check the pending heal entries from any node: ```bash sudo gluster volume heal gv0 info ``` ```output Brick gluster1:/data/glusterfs/gv0/brick Status: Connected Number of entries: 0 Brick gluster2:/data/glusterfs/gv0/brick Status: Connected Number of entries: 0 Brick gluster3:/data/glusterfs/gv0/brick Status: Connected Number of entries: 0 ``` Right after the node returns you may see a few entries under gluster1 and gluster2. They drop to `0` once healing finishes, and `ls /data/glusterfs/gv0/brick | wc -l` on gluster3 then returns `30`. ## Step 9 - Expanding the volume A replicated volume grows in sets of bricks equal to the replica count. To double the capacity of `gv0`, add one new brick on each of three nodes (new nodes or new disks on the existing ones), then rebalance so existing files spread across both sets: ```bash sudo gluster volume add-brick gv0 \ gluster1:/data/glusterfs/gv0b/brick \ gluster2:/data/glusterfs/gv0b/brick \ gluster3:/data/glusterfs/gv0b/brick sudo gluster volume rebalance gv0 start ``` Follow the progress until every node reports `completed`: ```bash sudo gluster volume rebalance gv0 status ``` The volume type changes to `Distributed-Replicate` with `2 x 3 = 6` bricks. Prepare each new brick exactly as in Step 2 before adding it. ## Troubleshooting **`peer probe: failed: Probe returned with Transport endpoint is not connected`.** The other node is not reachable on port 24007. Check that `glusterd` is running on it, that the hostname resolves to the private IP, and that the UFW rule from Step 4 exists on both sides. **`volume create: gv0: failed: The brick ... is being created in the root partition`.** The brick path is not on the dedicated disk, usually because the disk did not mount. Run `findmnt /data/glusterfs/gv0` and fix the mount instead of adding `force`. **The client mount hangs or fails.** Read the client log, which is named after the mount point: ```bash sudo tail -n 50 /var/log/glusterfs/mnt-gv0.log ``` Errors about brick ports mean the client cannot reach the 49152 and higher ports on some node. Errors about `auth.allow` mean the client IP is not in the allowed range. **Heal entries do not go down.** Check for split-brain files, which need manual resolution: ```bash sudo gluster volume heal gv0 info split-brain ``` Server logs are in `/var/log/glusterfs/glusterd.log`, and each brick has its own log in `/var/log/glusterfs/bricks/`. ## Conclusion You now have a three-node GlusterFS cluster serving a replicated volume that keeps working when a node fails, mounted on a client with automatic failover for the volume layout. From here you can move application data such as uploads or shared web content onto `/mnt/gv0`, enable TLS for the management and I/O paths if the traffic crosses untrusted networks, and monitor `gluster volume status` and `gluster volume heal gv0 info` from your monitoring system.