--- title: "How to Deploy a Ceph Cluster with cephadm on Ubuntu 24.04" description: "Deploy a three-node Ceph storage cluster with cephadm on Ubuntu 24.04, add OSDs, then use it for RBD block devices, a CephFS file system and S3 object storage." category: "Storage and Backup Advanced" language: en source: https://cubepath.com/docs/storage-and-backup-advanced/ceph-storage-cluster-installation updated: 2026-09-25 --- # How to Deploy a Ceph Cluster with cephadm on Ubuntu 24.04 Ceph is a distributed storage system that spreads data across many servers and disks, replicates it, and recovers automatically when a disk or node fails. A single Ceph cluster can provide block devices (RBD), a shared POSIX file system (CephFS) and S3-compatible object storage (RGW). In this tutorial you will deploy a three-node cluster on Ubuntu 24.04 with `cephadm`, the official deployment tool, add disks as OSDs, and try each of the three storage interfaces. ## Prerequisites To follow this guide you need three servers running Ubuntu 24.04 LTS, for example CubePath VPS or dedicated servers, each with: - At least 4 vCPU and 8 GB of RAM. Each OSD daemon aims to use about 4 GB of memory by default, and the monitor and manager daemons need memory too. - At least one additional, completely empty disk (no partitions, no file system, no LVM) for Ceph. Ceph will not use the system disk. - A private network between the nodes. This guide uses `10.0.0.11`, `10.0.0.12` and `10.0.0.13` for the hosts `ceph1`, `ceph2` and `ceph3`. - Root access over SSH from `ceph1` to the other two nodes, because cephadm manages the hosts as `root` over SSH. Replace the host names and addresses with your own throughout. ## How the cluster is laid out cephadm runs every Ceph daemon in a container managed by systemd. The main daemon types are: | Daemon | Role | In this cluster | | --- | --- | --- | | MON (monitor) | Holds the cluster map and forms a quorum | 3, one per node | | MGR (manager) | Orchestration, dashboard, metrics | 2, one active and one standby | | OSD | Stores data on one disk each | One per empty disk | | MDS | Metadata for CephFS | 2, created in Step 7 | | RGW | S3 and Swift gateway | 1, created in Step 8 | With three nodes, pools keep three copies of each object (one per host) and stay writable if one node goes down. ## Step 1 - Preparing all three nodes Run the commands in this step on **every** node. First, make the host names resolve on each node by adding them to `/etc/hosts`: ```bash sudo nano /etc/hosts ``` ```text 10.0.0.11 ceph1 10.0.0.12 ceph2 10.0.0.13 ceph3 ``` Set each node's short host name to match, for example on the first node: ```bash sudo hostnamectl set-hostname ceph1 ``` cephadm needs a container runtime, LVM for the OSDs and synchronized clocks, because monitors refuse to form a quorum when clocks drift. Install them: ```bash sudo apt update sudo apt install podman lvm2 chrony ``` Check that the clock is synchronized: ```bash timedatectl | grep synchronized ``` ```output System clock synchronized: yes ``` Allow all traffic between the nodes on the private network. Ceph uses ports 3300 and 6789 for monitors and the range 6800 to 7300 for OSDs and other daemons, so allowing the whole subnet is simpler and keeps the public interface closed: ```bash sudo ufw allow from 10.0.0.0/24 ``` Finally, identify the empty disk that Ceph will use: ```bash lsblk ``` ```output NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS vda 253:0 0 40G 0 disk ├─vda1 253:1 0 39G 0 part / ... vdb 253:16 0 100G 0 disk ``` Here `vdb` has no partitions or mount points, so it is available. Your disk name may differ. ## Step 2 - Installing cephadm and bootstrapping the first node On `ceph1` only, install `cephadm` and the Ceph command-line tools from the Ubuntu repositories. On Ubuntu 24.04 these are Ceph 19 (Squid), and cephadm will deploy the matching container images: ```bash sudo apt install cephadm ceph-common ``` Bootstrap the cluster. This creates the first monitor and manager on `ceph1`, generates the cluster SSH key and the admin keyring, and starts the dashboard: ```bash sudo cephadm bootstrap --mon-ip 10.0.0.11 ``` The command takes a few minutes while it pulls the container image. It ends with the dashboard credentials: ```output Ceph Dashboard is now available at: URL: https://ceph1:8443/ User: admin Password: 5xk0q2l7ab Enabling client.admin keyring and conf on hosts with "admin" label ... Bootstrap complete. ``` Save the password; you will be asked to change it on first login. Check the cluster from the host: ```bash sudo ceph -s ``` ```output cluster: id: 2c6b7e7a-9a1f-11f1-8f4e-525400a1b2c3 health: HEALTH_WARN OSD count 0 < osd_pool_default_size 3 services: mon: 1 daemons, quorum ceph1 (age 2m) mgr: ceph1.kxqvbz(active, since 1m) osd: 0 osds: 0 up, 0 in ``` The warning is expected: there are no OSDs yet. ## Step 3 - Adding the other hosts cephadm connects to the other nodes as `root` using the public key it generated in `/etc/ceph/ceph.pub`. Copy it to `ceph2` and `ceph3`: ```bash sudo ssh-copy-id -f -i /etc/ceph/ceph.pub root@ceph2 sudo ssh-copy-id -f -i /etc/ceph/ceph.pub root@ceph3 ``` If password login for root is disabled, append the contents of `/etc/ceph/ceph.pub` to `/root/.ssh/authorized_keys` on each node instead. Add the hosts to the cluster, with the `_admin` label so they also receive the configuration and admin keyring: ```bash sudo ceph orch host add ceph2 10.0.0.12 --labels _admin sudo ceph orch host add ceph3 10.0.0.13 --labels _admin sudo ceph orch host ls ``` ```output HOST ADDR LABELS STATUS ceph1 10.0.0.11 _admin ceph2 10.0.0.12 _admin ceph3 10.0.0.13 _admin 3 hosts in cluster ``` cephadm automatically deploys additional monitors and a standby manager on the new hosts. After a minute or two, confirm there are three monitors in quorum: ```bash sudo ceph mon stat ``` ```output e3: 3 mons at {ceph1=[v2:10.0.0.11:3300/0,v1:10.0.0.11:6789/0],ceph2=[...],ceph3=[...]} ... quorum 0,1,2 ceph1,ceph2,ceph3 ``` ## Step 4 - Creating OSDs List the disks that cephadm sees on all hosts. Only disks with `AVAILABLE` set to `Yes` can become OSDs: ```bash sudo ceph orch device ls ``` ```output HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS ceph1 /dev/vdb hdd 100G Yes 2m ago ceph2 /dev/vdb hdd 100G Yes 1m ago ceph3 /dev/vdb hdd 100G Yes 1m ago ``` Tell the orchestrator to turn every available disk into an OSD. This also applies to disks you add later: ```bash sudo ceph orch apply osd --all-available-devices ``` If you prefer to choose disks explicitly, add them one at a time instead with `sudo ceph orch daemon add osd ceph1:/dev/vdb`. After a few minutes the OSDs are up. Check the tree and the cluster health: ```bash sudo ceph osd tree sudo ceph -s ``` ```output ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 0.29306 root default -3 0.09769 host ceph1 0 hdd 0.09769 osd.0 up 1.00000 1.00000 -5 0.09769 host ceph2 1 hdd 0.09769 osd.1 up 1.00000 1.00000 -7 0.09769 host ceph3 2 hdd 0.09769 osd.2 up 1.00000 1.00000 cluster: health: HEALTH_OK ``` A disk that shows `Insufficient space` or `Has partitions` as the reject reason still contains old data. After double-checking that it is the right disk, wipe it with `sudo ceph orch device zap ceph2 /dev/vdb --force`. ## Step 5 - Accessing the dashboard The dashboard listens on port 8443 of the active manager. Rather than opening it to the internet, reach it through an SSH tunnel from your workstation: ```bash ssh -L 8443:10.0.0.11:8443 your_user@ceph1_public_ip ``` Then open `https://localhost:8443` in your browser, accept the self-signed certificate and log in as `admin` with the password from Step 2. If the manager fails over to another node, `sudo ceph mgr services` shows the current dashboard URL. ## Step 6 - Using block storage with RBD RBD provides virtual disks stored in Ceph, used by Proxmox, OpenStack, Kubernetes and plain Linux hosts. Create a pool for them and initialize it. The placement group autoscaler is on by default, so you do not need to choose a PG count: ```bash sudo ceph osd pool create rbd sudo rbd pool init rbd ``` Create a 10 GB image and map it as a block device on `ceph1`: ```bash sudo rbd create --size 10G rbd/test-disk sudo rbd map rbd/test-disk ``` ```output /dev/rbd0 ``` Format and mount it like any other disk: ```bash sudo mkfs.ext4 /dev/rbd0 sudo mkdir -p /mnt/rbd sudo mount /dev/rbd0 /mnt/rbd df -h /mnt/rbd ``` ```output Filesystem Size Used Avail Use% Mounted on /dev/rbd0 9.8G 24K 9.3G 1% /mnt/rbd ``` When you are done testing, unmount and unmap it with `sudo umount /mnt/rbd && sudo rbd unmap /dev/rbd0`. ## Step 7 - Creating a CephFS file system CephFS is a shared file system that many clients can mount at the same time. The `ceph fs volume create` command creates the data and metadata pools and asks the orchestrator to deploy MDS daemons: ```bash sudo ceph fs volume create cephfs sudo ceph fs status cephfs ``` ```output cephfs - 0 clients ====== RANK STATE MDS ACTIVITY DNS INOS DIRS CAPS 0 active cephfs.ceph2.hqzvkt Reqs: 0 /s 10 13 12 0 POOL TYPE USED AVAIL cephfs.cephfs.meta metadata 96.0k 94.9G cephfs.cephfs.data data 0 94.9G STANDBY MDS cephfs.ceph3.jmwpfa ``` Create a client key that can only access this file system, instead of using the admin key: ```bash sudo ceph fs authorize cephfs client.fsuser / rw | sudo tee /etc/ceph/ceph.client.fsuser.keyring sudo chmod 600 /etc/ceph/ceph.client.fsuser.keyring ``` Mount it with the kernel client. The `fsuser@.cephfs=/` syntax means user `fsuser`, the cluster in `/etc/ceph/ceph.conf`, file system `cephfs`, path `/`: ```bash sudo mkdir -p /mnt/cephfs sudo mount -t ceph fsuser@.cephfs=/ /mnt/cephfs df -h /mnt/cephfs ``` ```output Filesystem Size Used Avail Use% Mounted on 10.0.0.11:6789,...:/ 95G 0 95G 0% /mnt/cephfs ``` To mount it on another machine, install `ceph-common` there and copy `/etc/ceph/ceph.conf` and the `fsuser` keyring to its `/etc/ceph` directory. ## Step 8 - Providing S3 object storage with RGW The RADOS Gateway exposes Ceph through the S3 API. Deploy one gateway on `ceph1` listening on port 8080: ```bash sudo ceph orch apply rgw s3 --placement="ceph1" --port=8080 sudo ceph orch ps --daemon-type rgw ``` ```output NAME HOST PORTS STATUS REFRESHED AGE rgw.s3.ceph1.xbmwqd ceph1 *:8080 running (30s) 20s ago 31s ``` Create an S3 user. `radosgw-admin` runs inside the Ceph container, which `cephadm shell` opens for you: ```bash sudo cephadm shell -- radosgw-admin user create --uid=appuser --display-name="App user" ``` The JSON output contains the credentials: ```output "keys": [ { "user": "appuser", "access_key": "Q3V7J5N0H2X8KZ1LAB4C", "secret_key": "b7c2...your_secret_key" } ], ``` Test the endpoint with the AWS CLI or any S3 client. For example, with `s3cmd` from the Ubuntu repositories: ```bash sudo apt install s3cmd s3cmd --access_key=your_access_key --secret_key=your_secret_key \ --host=10.0.0.11:8080 --host-bucket=10.0.0.11:8080 --no-ssl mb s3://test-bucket ``` ```output Bucket 's3://test-bucket/' created ``` For production use, put the gateway behind a reverse proxy or load balancer with TLS rather than exposing plain HTTP. ## Troubleshooting **`ceph -s` shows `clock skew detected`.** A node's clock drifted. Check `chronyc tracking` on each node and make sure `chrony` is running and can reach its time servers. **A host shows as `Offline` in `ceph orch host ls`.** cephadm cannot SSH to it. Run `sudo ceph cephadm check-host ceph2` to see the exact error, usually a missing key in `/root/.ssh/authorized_keys` or a firewall rule. **`HEALTH_WARN` with `pools have too many placement groups` or `pool has no application enabled`.** The first is informational while the autoscaler adjusts. For the second, enable the application with `sudo ceph osd pool application enable rbd` (or `cephfs`, `rgw`). **An OSD is `down`.** Find the host with `sudo ceph osd tree` and check the daemon state with `sudo ceph orch ps --daemon-type osd`. Then, on that host, read its log with `sudo cephadm logs --name osd.1`. ## Conclusion You now have a three-node Ceph cluster managed by cephadm, with three-way replication across hosts and working block, file and object interfaces. Next steps are adding more disks or nodes (the OSD service picks up new empty disks automatically), moving replication traffic to a dedicated network (the `cluster_network` option, easiest to set with `--cluster-network` at bootstrap), and wiring the `rbd` pool into Proxmox or Kubernetes through Ceph CSI.