Ceph is a distributed storage system that spreads data across many servers and disks, replicates it, and recovers automatically when a disk or node fails. A single Ceph cluster can provide block devices (RBD), a shared POSIX file system (CephFS) and S3-compatible object storage (RGW). In this tutorial you will deploy a three-node cluster on Ubuntu 24.04 with cephadm, the official deployment tool, add disks as OSDs, and try each of the three storage interfaces.

Prerequisites

To follow this guide you need three servers running Ubuntu 24.04 LTS, for example CubePath VPS or dedicated servers, each with:

  • At least 4 vCPU and 8 GB of RAM. Each OSD daemon aims to use about 4 GB of memory by default, and the monitor and manager daemons need memory too.
  • At least one additional, completely empty disk (no partitions, no file system, no LVM) for Ceph. Ceph will not use the system disk.
  • A private network between the nodes. This guide uses 10.0.0.11, 10.0.0.12 and 10.0.0.13 for the hosts ceph1, ceph2 and ceph3.
  • Root access over SSH from ceph1 to the other two nodes, because cephadm manages the hosts as root over SSH.

Replace the host names and addresses with your own throughout.

How the cluster is laid out

cephadm runs every Ceph daemon in a container managed by systemd. The main daemon types are:

DaemonRoleIn this cluster
MON (monitor)Holds the cluster map and forms a quorum3, one per node
MGR (manager)Orchestration, dashboard, metrics2, one active and one standby
OSDStores data on one disk eachOne per empty disk
MDSMetadata for CephFS2, created in Step 7
RGWS3 and Swift gateway1, created in Step 8

With three nodes, pools keep three copies of each object (one per host) and stay writable if one node goes down.

Step 1 - Preparing all three nodes

Run the commands in this step on every node. First, make the host names resolve on each node by adding them to /etc/hosts:

sudo nano /etc/hosts

10.0.0.11 ceph1
10.0.0.12 ceph2
10.0.0.13 ceph3

Set each node's short host name to match, for example on the first node:

sudo hostnamectl set-hostname ceph1

cephadm needs a container runtime, LVM for the OSDs and synchronized clocks, because monitors refuse to form a quorum when clocks drift. Install them:

sudo apt update
sudo apt install podman lvm2 chrony

Check that the clock is synchronized:

timedatectl | grep synchronized

System clock synchronized: yes

Allow all traffic between the nodes on the private network. Ceph uses ports 3300 and 6789 for monitors and the range 6800 to 7300 for OSDs and other daemons, so allowing the whole subnet is simpler and keeps the public interface closed:

sudo ufw allow from 10.0.0.0/24

Finally, identify the empty disk that Ceph will use:

lsblk

NAME    MAJ:MIN RM  SIZE RO TYPE MOUNTPOINTS
vda     253:0    0   40G  0 disk
├─vda1  253:1    0   39G  0 part /
...
vdb     253:16   0  100G  0 disk

Here vdb has no partitions or mount points, so it is available. Your disk name may differ.

Step 2 - Installing cephadm and bootstrapping the first node

On ceph1 only, install cephadm and the Ceph command-line tools from the Ubuntu repositories. On Ubuntu 24.04 these are Ceph 19 (Squid), and cephadm will deploy the matching container images:

sudo apt install cephadm ceph-common

Bootstrap the cluster. This creates the first monitor and manager on ceph1, generates the cluster SSH key and the admin keyring, and starts the dashboard:

sudo cephadm bootstrap --mon-ip 10.0.0.11

The command takes a few minutes while it pulls the container image. It ends with the dashboard credentials:

Ceph Dashboard is now available at:

	     URL: https://ceph1:8443/
	    User: admin
	Password: 5xk0q2l7ab

Enabling client.admin keyring and conf on hosts with "admin" label
...
Bootstrap complete.

Save the password; you will be asked to change it on first login. Check the cluster from the host:

sudo ceph -s

  cluster:
    id:     2c6b7e7a-9a1f-11f1-8f4e-525400a1b2c3
    health: HEALTH_WARN
            OSD count 0 < osd_pool_default_size 3

  services:
    mon: 1 daemons, quorum ceph1 (age 2m)
    mgr: ceph1.kxqvbz(active, since 1m)
    osd: 0 osds: 0 up, 0 in

The warning is expected: there are no OSDs yet.

Step 3 - Adding the other hosts

cephadm connects to the other nodes as root using the public key it generated in /etc/ceph/ceph.pub. Copy it to ceph2 and ceph3:

sudo ssh-copy-id -f -i /etc/ceph/ceph.pub root@ceph2
sudo ssh-copy-id -f -i /etc/ceph/ceph.pub root@ceph3

If password login for root is disabled, append the contents of /etc/ceph/ceph.pub to /root/.ssh/authorized_keys on each node instead.

Add the hosts to the cluster, with the _admin label so they also receive the configuration and admin keyring:

sudo ceph orch host add ceph2 10.0.0.12 --labels _admin
sudo ceph orch host add ceph3 10.0.0.13 --labels _admin
sudo ceph orch host ls

HOST   ADDR       LABELS  STATUS
ceph1  10.0.0.11  _admin
ceph2  10.0.0.12  _admin
ceph3  10.0.0.13  _admin
3 hosts in cluster

cephadm automatically deploys additional monitors and a standby manager on the new hosts. After a minute or two, confirm there are three monitors in quorum:

sudo ceph mon stat

e3: 3 mons at {ceph1=[v2:10.0.0.11:3300/0,v1:10.0.0.11:6789/0],ceph2=[...],ceph3=[...]} ... quorum 0,1,2 ceph1,ceph2,ceph3

Step 4 - Creating OSDs

List the disks that cephadm sees on all hosts. Only disks with AVAILABLE set to Yes can become OSDs:

sudo ceph orch device ls

HOST   PATH      TYPE  DEVICE ID  SIZE  AVAILABLE  REFRESHED  REJECT REASONS
ceph1  /dev/vdb  hdd              100G  Yes        2m ago
ceph2  /dev/vdb  hdd              100G  Yes        1m ago
ceph3  /dev/vdb  hdd              100G  Yes        1m ago

Tell the orchestrator to turn every available disk into an OSD. This also applies to disks you add later:

sudo ceph orch apply osd --all-available-devices

If you prefer to choose disks explicitly, add them one at a time instead with sudo ceph orch daemon add osd ceph1:/dev/vdb.

After a few minutes the OSDs are up. Check the tree and the cluster health:

sudo ceph osd tree
sudo ceph -s

ID  CLASS  WEIGHT   TYPE NAME       STATUS  REWEIGHT  PRI-AFF
-1         0.29306  root default
-3         0.09769      host ceph1
 0    hdd  0.09769          osd.0       up   1.00000  1.00000
-5         0.09769      host ceph2
 1    hdd  0.09769          osd.1       up   1.00000  1.00000
-7         0.09769      host ceph3
 2    hdd  0.09769          osd.2       up   1.00000  1.00000

  cluster:
    health: HEALTH_OK

A disk that shows Insufficient space or Has partitions as the reject reason still contains old data. After double-checking that it is the right disk, wipe it with sudo ceph orch device zap ceph2 /dev/vdb --force.

Step 5 - Accessing the dashboard

The dashboard listens on port 8443 of the active manager. Rather than opening it to the internet, reach it through an SSH tunnel from your workstation:

ssh -L 8443:10.0.0.11:8443 your_user@ceph1_public_ip

Then open https://localhost:8443 in your browser, accept the self-signed certificate and log in as admin with the password from Step 2. If the manager fails over to another node, sudo ceph mgr services shows the current dashboard URL.

Step 6 - Using block storage with RBD

RBD provides virtual disks stored in Ceph, used by Proxmox, OpenStack, Kubernetes and plain Linux hosts. Create a pool for them and initialize it. The placement group autoscaler is on by default, so you do not need to choose a PG count:

sudo ceph osd pool create rbd
sudo rbd pool init rbd

Create a 10 GB image and map it as a block device on ceph1:

sudo rbd create --size 10G rbd/test-disk
sudo rbd map rbd/test-disk

/dev/rbd0

Format and mount it like any other disk:

sudo mkfs.ext4 /dev/rbd0
sudo mkdir -p /mnt/rbd
sudo mount /dev/rbd0 /mnt/rbd
df -h /mnt/rbd

Filesystem      Size  Used Avail Use% Mounted on
/dev/rbd0       9.8G   24K  9.3G   1% /mnt/rbd

When you are done testing, unmount and unmap it with sudo umount /mnt/rbd && sudo rbd unmap /dev/rbd0.

Step 7 - Creating a CephFS file system

CephFS is a shared file system that many clients can mount at the same time. The ceph fs volume create command creates the data and metadata pools and asks the orchestrator to deploy MDS daemons:

sudo ceph fs volume create cephfs
sudo ceph fs status cephfs

cephfs - 0 clients
======
RANK  STATE           MDS             ACTIVITY     DNS    INOS   DIRS   CAPS
 0    active  cephfs.ceph2.hqzvkt  Reqs:    0 /s    10     13     12      0
      POOL           TYPE     USED  AVAIL
cephfs.cephfs.meta  metadata  96.0k  94.9G
cephfs.cephfs.data    data       0   94.9G
    STANDBY MDS
cephfs.ceph3.jmwpfa

Create a client key that can only access this file system, instead of using the admin key:

sudo ceph fs authorize cephfs client.fsuser / rw | sudo tee /etc/ceph/ceph.client.fsuser.keyring
sudo chmod 600 /etc/ceph/ceph.client.fsuser.keyring

Mount it with the kernel client. The [email protected]=/ syntax means user fsuser, the cluster in /etc/ceph/ceph.conf, file system cephfs, path /:

sudo mkdir -p /mnt/cephfs
sudo mount -t ceph [email protected]=/ /mnt/cephfs
df -h /mnt/cephfs

Filesystem              Size  Used Avail Use% Mounted on
10.0.0.11:6789,...:/     95G     0   95G   0% /mnt/cephfs

To mount it on another machine, install ceph-common there and copy /etc/ceph/ceph.conf and the fsuser keyring to its /etc/ceph directory.

Step 8 - Providing S3 object storage with RGW

The RADOS Gateway exposes Ceph through the S3 API. Deploy one gateway on ceph1 listening on port 8080:

sudo ceph orch apply rgw s3 --placement="ceph1" --port=8080
sudo ceph orch ps --daemon-type rgw

NAME                   HOST   PORTS   STATUS         REFRESHED  AGE
rgw.s3.ceph1.xbmwqd    ceph1  *:8080  running (30s)  20s ago    31s

Create an S3 user. radosgw-admin runs inside the Ceph container, which cephadm shell opens for you:

sudo cephadm shell -- radosgw-admin user create --uid=appuser --display-name="App user"

The JSON output contains the credentials:

    "keys": [
        {
            "user": "appuser",
            "access_key": "Q3V7J5N0H2X8KZ1LAB4C",
            "secret_key": "b7c2...your_secret_key"
        }
    ],

Test the endpoint with the AWS CLI or any S3 client. For example, with s3cmd from the Ubuntu repositories:

sudo apt install s3cmd
s3cmd --access_key=your_access_key --secret_key=your_secret_key \
  --host=10.0.0.11:8080 --host-bucket=10.0.0.11:8080 --no-ssl mb s3://test-bucket

Bucket 's3://test-bucket/' created

For production use, put the gateway behind a reverse proxy or load balancer with TLS rather than exposing plain HTTP.

Troubleshooting

ceph -s shows clock skew detected. A node's clock drifted. Check chronyc tracking on each node and make sure chrony is running and can reach its time servers.

A host shows as Offline in ceph orch host ls. cephadm cannot SSH to it. Run sudo ceph cephadm check-host ceph2 to see the exact error, usually a missing key in /root/.ssh/authorized_keys or a firewall rule.

HEALTH_WARN with pools have too many placement groups or pool has no application enabled. The first is informational while the autoscaler adjusts. For the second, enable the application with sudo ceph osd pool application enable <pool> rbd (or cephfs, rgw).

An OSD is down. Find the host with sudo ceph osd tree and check the daemon state with sudo ceph orch ps --daemon-type osd. Then, on that host, read its log with sudo cephadm logs --name osd.1.

Conclusion

You now have a three-node Ceph cluster managed by cephadm, with three-way replication across hosts and working block, file and object interfaces. Next steps are adding more disks or nodes (the OSD service picks up new empty disks automatically), moving replication traffic to a dedicated network (the cluster_network option, easiest to set with --cluster-network at bootstrap), and wiring the rbd pool into Proxmox or Kubernetes through Ceph CSI.