Ceph Storage Cluster Installation

Ceph is a unified, distributed storage system designed to provide excellent performance, reliability, and scalability for enterprise data centers. Unlike traditional SAN architectures, Ceph scales horizontally and eliminates single points of failure through intelligent data replication and distribution. This comprehensive guide covers deploying a production-ready Ceph cluster using cephadm, configuring multiple storage services, and managing the cluster's advanced features.

Table of Contents

  1. Ceph Architecture Overview
  2. Prerequisites and Planning
  3. Initial Bootstrap with cephadm
  4. Monitor and Manager Configuration
  5. OSD Installation and Management
  6. Ceph Filesystems and RBD
  7. Object Gateway Configuration
  8. Cluster Tuning and Optimization
  9. Monitoring and Maintenance
  10. Conclusion

Ceph Architecture Overview

Ceph architecture consists of three primary service types:

  • Monitors (MONs): Maintain cluster maps and state, typically deployed in odd numbers (3, 5, or 7) for consensus
  • Managers (MGRs): Handle cluster management, dashboards, and orchestration, minimum 2 for HA
  • Object Storage Daemons (OSDs): Store data and handle data movement, replication, and recovery

Supporting services include:

  • Metadata Servers (MDSs): Enable CephFS capabilities
  • Ceph Object Gateway (RGW): Provides S3-compatible API
  • Block Device Manager: Enables RBD integration

Prerequisites and Planning

Hardware Requirements

For a production Ceph cluster with 3 storage nodes:

  • Each node: 8 vCPU, 16 GB RAM minimum, 32 GB recommended
  • Storage: 4+ dedicated drives per node (NVMe preferred for performance)
  • Network: 10 Gbps interconnect, separate public and cluster networks
  • OS: CentOS/RHEL 8+, Ubuntu 20.04 LTS or later

Network Planning

# Example network layout
# Public Network (client access): 192.168.1.0/24
# Cluster Network (inter-OSD): 192.168.100.0/24

# Check current networking
ip addr show
ip route show

Disk Preparation

Identify available storage devices:

# List all block devices
lsblk

# Check disk health
sudo smartctl -a /dev/nvme0n1

# Wipe any existing filesystem (destructive)
sudo wipefs -a /dev/nvme0n1p1

Initial Bootstrap with cephadm

Installing cephadm

cephadm is Ceph's modern deployment tool, containerizing all Ceph components:

# Install cephadm on first node
curl --silent --remote-name --location https://github.com/ceph/ceph/raw/octopus/src/cephadm/cephadm
chmod +x cephadm
sudo ./cephadm add-repo --release octopus

# Install required dependencies
sudo apt-get install -y docker.io podman

# Or with yum (CentOS/RHEL)
sudo yum install -y podman

# Start container runtime
sudo systemctl enable podman
sudo systemctl start podman

Bootstrap the Cluster

Initialize the first monitor and manager:

# Run bootstrap command
sudo ./cephadm bootstrap \
  --mon-ip 192.168.1.10 \
  --cluster-network 192.168.100.0/24 \
  --allow-fqdn-hostname

# This creates:
# - First monitor at 192.168.1.10
# - Bootstrap manager and dashboard
# - /etc/ceph/ceph.conf
# - /etc/ceph/ceph.client.admin.keyring

# Verify bootstrap completed
sudo cephadm ls

# Get initial admin password (stored in dashboard)
sudo cat /var/lib/ceph/*/passwd-*

Install ceph-common on All Nodes

# Add Ceph repository
sudo ./cephadm add-repo --release octopus

# Install client tools
sudo apt-get install -y ceph-common ceph-base

# Or with yum
sudo yum install -y ceph-common ceph-base

# Verify installation
ceph --version

Monitor and Manager Configuration

Adding Additional Monitors

For high availability, deploy odd numbers of monitors (3, 5, or 7):

# List current monitors
sudo ceph mon stat

# Add monitor on second node
sudo cephadm shell ceph orch host add node2 192.168.1.11
sudo ceph orch daemon add mon node2:192.168.1.11

# Add monitor on third node
sudo cephadm shell ceph orch host add node3 192.168.1.12
sudo ceph orch daemon add mon node3:192.168.1.12

# Wait for election (usually 30-60 seconds)
sudo ceph mon stat

# Verify monitor cluster
sudo ceph quorum_status

Manager Configuration

Deploy multiple managers for automatic failover:

# List current managers
sudo ceph mgr stat

# Add manager on existing nodes
sudo ceph orch daemon add mgr node2
sudo ceph orch daemon add mgr node3

# Enable manager modules
sudo ceph mgr module enable dashboard
sudo ceph mgr module enable prometheus
sudo ceph mgr module enable telemetry

# Generate self-signed certificate for dashboard
sudo ceph dashboard create-self-signed-cert

# Get dashboard login credentials
sudo ceph dashboard get-admin-credentials

Dashboard Access

# Get active manager hostname and port
sudo ceph mgr services

# Access at https://<manager-ip>:8443/
# Login with credentials from previous command

# Or access via CLI
sudo ceph dashboard set-login-credentials admin password123

OSD Installation and Management

Discovering Available Disks

# List disks available on all nodes
sudo cephadm shell ceph orch device ls

# Check disk inventory
sudo cephadm shell ceph orch device ls --format json

# Show available devices on specific host
sudo cephadm shell ceph orch device ls node2

Creating OSDs

Cephadm automatically provisions OSDs from available devices:

# Enable ceph-volume for automatic OSD creation
sudo cephadm shell ceph orch apply osd --all-available-devices

# Or create OSD on specific device
sudo cephadm shell ceph orch daemon add osd node1:/dev/nvme0n1
sudo cephadm shell ceph orch daemon add osd node2:/dev/nvme0n1
sudo cephadm shell ceph orch daemon add osd node3:/dev/nvme0n1

# Monitor OSD creation progress
sudo ceph osd tree

# Check OSD status
sudo ceph -s

# Verify all OSDs are up and in
sudo ceph osd status

OSD Management

# List OSDs with detailed information
sudo ceph osd tree

# Check individual OSD status
sudo ceph osd dump

# Set OSD flags
sudo ceph osd set noout          # Prevent rebalancing during maintenance
sudo ceph osd unset noout        # Resume rebalancing

# Reweight OSD (for traffic distribution)
sudo ceph osd reweight 0 0.8     # Reduce OSD 0 to 80% weight

# Replace failed OSD
sudo ceph osd out osd.0          # Mark OSD as out
sudo ceph osd crush remove osd.0 # Remove from CRUSH
sudo ceph osd rm osd.0           # Delete OSD
sudo cephadm shell ceph orch daemon add osd node1:/dev/replacement_device

Ceph Filesystems and RBD

Creating CephFS (Metadata Server Pool Setup)

# Create metadata pool
sudo ceph osd pool create cephfs_metadata 32 32

# Create data pool
sudo ceph osd pool create cephfs_data 64 64

# Create CephFS
sudo ceph fs new cephfs cephfs_metadata cephfs_data

# Deploy metadata servers
sudo ceph orch daemon add mds cephfs node1
sudo ceph orch daemon add mds cephfs node2

# Verify CephFS
sudo ceph fs status
sudo ceph mds stat

Mounting CephFS

On client systems:

# Install CephFS client
sudo apt-get install -y ceph-fuse

# Create mount point
sudo mkdir -p /mnt/cephfs

# Get client keyring (from admin node)
sudo scp root@ceph-node1:/etc/ceph/ceph.client.admin.keyring /etc/ceph/

# Mount using fuse
sudo ceph-fuse -m 192.168.1.10 /mnt/cephfs

# Mount using kernel driver (recommended)
sudo mount -t ceph 192.168.1.10:/ /mnt/cephfs -o name=admin

# Verify mount
df -h /mnt/cephfs

RBD (Rados Block Device) Configuration

# Create RBD pool
sudo ceph osd pool create rbd 64 64

# Enable rbd application
sudo ceph osd pool application enable rbd rbd

# Create RBD image
sudo rbd create --size 10GB myimage --pool rbd

# List RBD images
sudo rbd ls rbd

# Map RBD to local device
sudo rbd map rbd/myimage

# Format and mount RBD
sudo mkfs.ext4 /dev/rbd0
sudo mount /dev/rbd0 /mnt/rbd

# Verify mount
df -h /mnt/rbd

# Unmap RBD when done
sudo rbd unmap /dev/rbd0

Object Gateway Configuration

Deploying Ceph Object Gateway (RGW)

# Create RGW pool
sudo ceph osd pool create .rgw.root 8 8
sudo ceph osd pool create default.rgw.log 8 8
sudo ceph osd pool create default.rgw.control 8 8
sudo ceph osd pool create default.rgw.meta 8 8
sudo ceph osd pool create default.rgw.data.root 8 8

# Deploy RGW daemon
sudo ceph orch daemon add rgw myrgw --placement="3 node1,node2,node3"

# Access RGW endpoint
sudo ceph orch ls | grep rgw

# Get RGW port information
sudo ss -tlnp | grep radosgw

Creating RGW Users

# Create S3 user
sudo radosgw-admin user create --uid=myappuser --display-name="My App User"

# Get access and secret keys
sudo radosgw-admin user info --uid=myappuser

# Create S3 subuser
sudo radosgw-admin subuser create --uid=myappuser --subuser=myappuser:swift

# Modify user quota
sudo radosgw-admin quota set --quota-scope=user --uid=myappuser --max-size=1T

# Suspend user
sudo radosgw-admin user suspend --uid=myappuser

# Remove user
sudo radosgw-admin user rm --uid=myappuser

Accessing Object Storage

# Configure s3cmd
sudo apt-get install -y s3cmd

# Setup s3cmd
s3cmd --configure

# When prompted:
# - Access Key: (from radosgw-admin output)
# - Secret Key: (from radosgw-admin output)
# - S3 Endpoint: your-rgw-host:7480

# Test S3 access
s3cmd ls
s3cmd mb s3://my-bucket
s3cmd put /tmp/file.txt s3://my-bucket/
s3cmd ls s3://my-bucket/

Cluster Tuning and Optimization

CRUSH Map Modification

The CRUSH map controls data placement:

# Get current CRUSH map
sudo ceph osd getcrushmap -o /tmp/crushmap.bin

# Decompile to readable format
sudo crushtool -d /tmp/crushmap.bin -o /tmp/crushmap.txt

# View CRUSH map structure
cat /tmp/crushmap.txt

# Recompile after modifications
sudo crushtool -c /tmp/crushmap.txt -o /tmp/crushmap-new.bin

# Apply new CRUSH map
sudo ceph osd setcrushmap -i /tmp/crushmap-new.bin

# Set replication rules
sudo ceph osd pool set rbd crush_rule replicated_rule

Performance Tuning

# Adjust recovery speed
sudo ceph tell osd.* config set osd_recovery_max_active 5
sudo ceph tell osd.* config set osd_recovery_sleep 0.5

# Tune journal settings
sudo ceph config set osd journal_size 5368

# Adjust PG count (set to next power of 2)
sudo ceph osd pool set mypool pg_num 128
sudo ceph osd pool set mypool pgp_num 128

# Monitor performance
sudo ceph -w
sudo ceph pg stat

Backfill and Rebalancing

# Monitor backfill status
sudo ceph status

# Set backfill priority
sudo ceph tell osd.* config set osd_backfill_scan_min 32
sudo ceph tell osd.* config set osd_backfill_scan_max 4096

# Pause recovery operations
sudo ceph osd set pause
sudo ceph osd set norebalance

# Resume operations
sudo ceph osd unset pause
sudo ceph osd unset norebalance

Monitoring and Maintenance

Health Monitoring

# Get cluster health status
sudo ceph health

# Detailed health information
sudo ceph health detail

# Monitor cluster status continuously
sudo ceph -w

# Check OSD fill levels
sudo ceph osd df

# Get cluster usage statistics
sudo ceph df

Cluster Logs

# View recent logs
sudo ceph log last 50

# Monitor logs in real-time
sudo ceph log tail

# Search logs for errors
sudo ceph log last 1000 | grep -i "error\|warn"

Backup and Recovery

# Backup cluster configuration
sudo ceph config dump > /backup/ceph-config.txt

# Backup pool metadata
sudo ceph osd pool get-all > /backup/pools.txt

# Backup CRUSH map
sudo ceph osd getcrushmap -o /backup/crushmap.bin

# Restore from backup
sudo ceph osd setcrushmap -i /backup/crushmap.bin

Conclusion

Ceph provides a powerful, scalable storage solution suitable for cloud environments, HPC centers, and enterprise data centers. By following this guide's procedures for deployment with cephadm, configuring multiple service types, and applying performance optimizations, you establish a robust foundation for diverse storage workloads. Regular monitoring through the dashboard and CLI tools ensures your cluster maintains optimal performance and data reliability. Whether deploying block storage, object storage, or shared filesystems, Ceph's unified architecture delivers the flexibility and performance modern infrastructure demands.