Disk I/O Benchmarking with fio

fio (Flexible I/O Tester) is the industry-standard storage benchmarking tool providing precise control over workload characteristics, making it essential for storage infrastructure validation and optimization. With support for sequential and random workloads, multiple job configurations, and detailed performance metrics, fio enables comprehensive storage performance analysis. This guide covers fio installation, common benchmarks, and result interpretation.

fio Installation and Basics
Sequential Workload Testing
Random I/O Benchmarking
IOPS and Latency Analysis
Job Files and Configuration
Storage Device Comparison
Advanced Workload Simulation
Performance Analysis and Reporting
Conclusion

fio Installation and Basics

Installing fio

# Ubuntu/Debian installation
sudo apt-get update
sudo apt-get install -y fio

# CentOS/RHEL installation
sudo yum install -y fio

# Compile from latest source
git clone https://github.com/axboe/fio.git
cd fio
./configure
make
sudo make install

# Verify installation
fio --version

Storage Device Preparation

# Identify target device
lsblk
sudo fdisk -l

# Check current device usage
df -h
sudo lsof | grep /dev/sda

# For testing (unmounted device only)
sudo fio --filename=/dev/sdb --rw=read --bs=4k --iodepth=32 \
  --runtime=10 --group_reporting --name=random_read

# For testing partition
sudo fio --filename=/mnt/test/fio_test_file --rw=read --bs=4k \
  --iodepth=32 --runtime=10 --size=10G --name=file_read

Sequential Workload Testing

Sequential Read Performance

# Baseline sequential read
fio --filename=/mnt/test/fio_test_file --rw=read --bs=1m \
  --iodepth=32 --runtime=30 --name=seq_read

# Parameters explained:
# --filename: Test file location
# --rw=read: Sequential read
# --bs: Block size (1m = 1 MB)
# --iodepth: Queue depth (32 typical)
# --runtime: Test duration (seconds)

# Large block sequential (streaming)
fio --filename=/mnt/test/fio_test_file --rw=read --bs=4m \
  --iodepth=16 --runtime=60 --size=100G --name=streaming_read

# Typical results (SSD):
# read: IOPS=1234.56, BW=1240.56 MiB/s
# Throughput: ~1.2 GB/s

Sequential Write Performance

# Sequential write test
fio --filename=/mnt/test/fio_test_file --rw=write --bs=1m \
  --iodepth=32 --runtime=30 --name=seq_write --pre=touch \
  --nrfiles=1 --filesize=100G

# Fill storage with sequential write (cache flush test)
fio --filename=/mnt/test/fio_test_file --rw=write --bs=1m \
  --iodepth=64 --runtime=120 --size=500G --sync=1 --name=seq_write_sync

# Monitor write performance
watch -n 1 'iostat -x -d sdb 1'

Random I/O Benchmarking

Random Read IOPS

# Measure random read IOPS
fio --filename=/mnt/test/fio_test_file --rw=randread --bs=4k \
  --iodepth=32 --runtime=60 --size=10G --name=random_read_iops

# Expected results:
# SSD: 10,000-100,000+ IOPS
# HDD: 100-200 IOPS

# Increase queue depth for higher IOPS potential
fio --filename=/mnt/test/fio_test_file --rw=randread --bs=4k \
  --iodepth=128 --runtime=60 --size=10G --name=high_queue_depth

# Test with multiple file size
fio --filename=/mnt/test/fio_test_file --rw=randread --bs=4k \
  --iodepth=64 --runtime=30 --size=100G --name=large_working_set

Random Write IOPS

# Random write benchmark
fio --filename=/mnt/test/fio_test_file --rw=randwrite --bs=4k \
  --iodepth=32 --runtime=60 --size=10G --name=random_write_iops

# Sync writes (journal-like, important for databases)
fio --filename=/mnt/test/fio_test_file --rw=randwrite --bs=4k \
  --iodepth=16 --fsync=1 --runtime=60 --size=10G --name=sync_writes

# Monitor disk during test
iostat -x 1

# Test with direct I/O (bypasses cache)
fio --filename=/mnt/test/fio_test_file --rw=randwrite --bs=4k \
  --iodepth=32 --direct=1 --runtime=60 --size=10G --name=direct_writes

IOPS and Latency Analysis

Latency Percentiles

# Measure latency distribution
fio --filename=/mnt/test/fio_test_file --rw=randread --bs=4k \
  --iodepth=32 --runtime=60 --size=10G --lat_log=latency_results \
  --name=latency_test

# Analyze latency percentiles
# Output shows: min, max, mean, p50, p95, p99, p99.9

# Example latency results:
# lat (usec) : min=5, max=45000, avg=150, stdev=500
# percentiles (usec) :
#  50.00th=[100]
#  90.00th=[200]
#  99.00th=[500]
#  99.90th=[1500]
#  99.99th=[10000]

# Interpret results:
# p99 = 99% of requests complete within latency
# p99.9 = 99.9% of requests (tail latency)

Latency Distribution Analysis

# Generate detailed latency histogram
fio --filename=/mnt/test/fio_test_file --rw=randread --bs=4k \
  --iodepth=32 --runtime=120 --size=10G --log_histogram \
  --name=detailed_latency

# Test latency consistency over time
fio --filename=/mnt/test/fio_test_file --rw=randread --bs=4k \
  --iodepth=32 --runtime=300 --size=10G --lat_log=hourly_latency \
  --name=consistency_test

# Analyze for outliers
# High p99.9 indicates occasional latency spikes
# Suggests thermal throttling or garbage collection events

Job Files and Configuration

Creating fio Job Files

# Create reusable job file
cat > ~/storage_benchmark.fio <<'EOF'
[global]
ioengine=libaio
direct=1
group_reporting=1
time_based=1

[random_read]
rw=randread
bs=4k
iodepth=32
runtime=60
size=10G
numjobs=1

[random_write]
rw=randwrite
bs=4k
iodepth=32
runtime=60
size=10G
numjobs=1

[sequential_read]
rw=read
bs=1m
iodepth=16
runtime=60
size=100G
numjobs=1

[sequential_write]
rw=write
bs=1m
iodepth=16
runtime=60
size=100G
numjobs=1
EOF

# Run job file
fio ~/storage_benchmark.fio

Multi-Job Configuration

# Create comprehensive test suite
cat > ~/complete_benchmark.fio <<'EOF'
[global]
ioengine=libaio
direct=1
runtime=60
time_based=1
group_reporting=1
filename=/mnt/test/fio_test_file

[seq_read_4k]
rw=read
bs=4k
iodepth=32
stonewall

[seq_read_1m]
rw=read
bs=1m
iodepth=16
stonewall

[rnd_read_4k]
rw=randread
bs=4k
iodepth=32
stonewall

[rnd_write_4k]
rw=randwrite
bs=4k
iodepth=32
stonewall

[mixed_read_write]
rw=randrw
rwmixread=70
bs=4k
iodepth=32
stonewall
EOF

# Run complete suite
fio ~/complete_benchmark.fio

Storage Device Comparison

SSD vs HDD Testing

# Test SSD performance
echo "=== SSD Performance ==="
fio --filename=/mnt/ssd/fio_test --rw=randread --bs=4k \
  --iodepth=32 --runtime=60 --size=10G --name=ssd_randread

# Test HDD performance
echo "=== HDD Performance ==="
fio --filename=/mnt/hdd/fio_test --rw=randread --bs=4k \
  --iodepth=32 --runtime=60 --size=10G --name=hdd_randread

# Sequential comparison
echo "=== SSD Sequential ==="
fio --filename=/mnt/ssd/fio_test --rw=read --bs=1m \
  --iodepth=32 --runtime=60 --size=100G --name=ssd_seq

echo "=== HDD Sequential ==="
fio --filename=/mnt/hdd/fio_test --rw=read --bs=1m \
  --iodepth=32 --runtime=60 --size=100G --name=hdd_seq

Drive Aging and Wear Analysis

# Monitor SSD endurance during testing
# Create script for wear monitoring
cat > monitor_ssd_wear.sh <<'EOF'
#!/bin/bash
DEVICE=$1

echo "=== Pre-Test SSD Status ==="
sudo smartctl -a $DEVICE | grep -E "Wear_Leveling|Program_Fail|Erase_Fail"

# Run sustained workload
fio --filename=/mnt/test/fio_test --rw=randwrite --bs=4k \
  --iodepth=32 --runtime=7200 --size=100G --name=endurance_test

echo "=== Post-Test SSD Status ==="
sudo smartctl -a $DEVICE | grep -E "Wear_Leveling|Program_Fail|Erase_Fail"
EOF

chmod +x monitor_ssd_wear.sh
./monitor_ssd_wear.sh /dev/sdb

Advanced Workload Simulation

Database Workload Simulation

# OLTP-like workload (mixed small block I/O)
cat > oltp_workload.fio <<'EOF'
[global]
ioengine=libaio
direct=1
group_reporting=1
runtime=300
time_based=1

[db_transactions]
rw=randrw
rwmixread=80
bs=16k
iodepth=32
numjobs=4
filename=/mnt/test/database_file
size=50G
EOF

fio oltp_workload.fio

Streaming/Archive Workload

# Streaming video or backup workload
cat > streaming_workload.fio <<'EOF'
[global]
ioengine=libaio
direct=1
group_reporting=1

[streaming_read]
rw=read
bs=4m
iodepth=8
runtime=300
numjobs=2
filename=/mnt/test/streaming_file
size=500G
EOF

fio streaming_workload.fio

Performance Analysis and Reporting

JSON Output and Analysis

# Generate JSON output for parsing
fio --filename=/mnt/test/fio_test_file --rw=randread --bs=4k \
  --iodepth=32 --runtime=60 --size=10G --output-format=json \
  > results.json --name=test

# Parse specific metrics
cat results.json | jq '.jobs[0].read.iops'
cat results.json | jq '.jobs[0].read.bw'
cat results.json | jq '.jobs[0].read.lat_ns.percentile."99.000000"'

Creating Baseline Comparisons

# Establish baseline
fio --filename=/mnt/test/fio_test_file --rw=randread --bs=4k \
  --iodepth=32 --runtime=120 --size=10G --output-format=json \
  > baseline.json --name=baseline

# Run after modifications
fio --filename=/mnt/test/fio_test_file --rw=randread --bs=4k \
  --iodepth=32 --runtime=120 --size=10G --output-format=json \
  > after_tuning.json --name=tuned

# Compare results
echo "Baseline IOPS: $(cat baseline.json | jq '.jobs[0].read.iops')"
echo "After tuning IOPS: $(cat after_tuning.json | jq '.jobs[0].read.iops')"

Conclusion

fio provides unmatched flexibility for storage performance validation, enabling accurate simulation of real-world workloads and detection of storage bottlenecks. By understanding block size effects, queue depth optimization, and latency percentile implications, infrastructure teams make informed storage purchasing and optimization decisions. Regular benchmarking establishes baselines detecting performance degradation from age, thermal throttling, or firmware issues. Whether optimizing database performance, validating new storage infrastructure, or troubleshooting I/O bottlenecks, fio remains the essential tool for storage engineering excellence.

Disk IO benchmarking with fio