Server Benchmarking with sysbench

Sysbench is a modular, cross-platform benchmarking tool providing comprehensive performance testing capabilities for modern server infrastructure. With tests for CPU, memory, I/O, and databases, sysbench enables data-driven capacity planning and performance validation. This guide covers using sysbench for thorough server benchmarking and interpreting results for infrastructure decisions.

Table of Contents

  1. Sysbench Overview and Installation
  2. CPU Benchmarking
  3. Memory Performance Testing
  4. File I/O Benchmarking
  5. Thread Contention Testing
  6. Database Benchmarking
  7. Results Interpretation
  8. Performance Baseline Comparison
  9. Conclusion

Sysbench Overview and Installation

System Requirements

# Check system specifications for baseline
uname -a
cat /proc/cpuinfo | head -20
free -h
nproc
lscpu

Installing Sysbench

# Ubuntu/Debian installation
sudo apt-get update
sudo apt-get install -y sysbench

# CentOS/RHEL installation
sudo yum install -y sysbench

# Or compile from source for latest version
git clone https://github.com/akopytov/sysbench.git
cd sysbench
./autogen.sh
./configure
make -j$(nproc)
sudo make install

# Verify installation
sysbench --version

# List available tests
sysbench --help

CPU Benchmarking

Basic CPU Test

# Run CPU benchmark on single thread
sysbench cpu --cpu-max-prime=20000 run

# Parameters explained:
# --cpu-max-prime: Largest prime number to check (higher = longer test)
# run: Execute the benchmark

# Example output interpretation:
# Total time: ~10 seconds
# Total number of events: Number of prime calculations completed
# Approximate throughput: Operations per second

Multi-threaded CPU Testing

# Test with multiple threads (match CPU core count)
sysbench cpu --threads=16 --cpu-max-prime=20000 run

# Recommended thread counts:
# - 1: Single-thread baseline
# - CPU core count: Full utilization
# - 2x CPU cores: Contention testing

# Compare single vs multi-threaded performance
echo "=== Single Thread ===" 
sysbench cpu --threads=1 --cpu-max-prime=20000 run

echo "=== 8 Threads ==="
sysbench cpu --threads=8 --cpu-max-prime=20000 run

echo "=== 16 Threads ==="
sysbench cpu --threads=16 --cpu-max-prime=20000 run

CPU Stress Testing

# Extended stress test (30 seconds)
sysbench cpu --threads=16 --time=30 --cpu-max-prime=50000 run

# Monitor while running
# In another terminal:
top
htop
sar 1

# Check thermal throttling
cat /proc/cpuinfo | grep MHz
watch -n 1 'cat /proc/cpuinfo | grep MHz'

# Long-running stability test (2 hours)
nohup sysbench cpu --threads=16 --time=7200 --cpu-max-prime=100000 run > cpu-long.log 2>&1 &
tail -f cpu-long.log

Memory Performance Testing

Memory Read/Write Benchmarking

# Test memory sequential read
sysbench memory --memory-total-size=10G run

# Test with different patterns
# Sequential access pattern
sysbench memory --memory-total-size=10G --memory-access-mode=seq run

# Random access pattern
sysbench memory --memory-total-size=10G --memory-access-mode=rnd run

# Specific block size
sysbench memory --memory-total-size=10G --memory-block-size=1K run
sysbench memory --memory-total-size=10G --memory-block-size=1M run

Memory Performance Analysis

# Compare different memory block sizes
for block_size in 1K 4K 8K 64K 256K 1M; do
  echo "=== Block Size: $block_size ==="
  sysbench memory --memory-total-size=10G --memory-block-size=$block_size run
done

# Multi-threaded memory test
sysbench memory --threads=8 --memory-total-size=20G run

# Monitor memory during test
# In parallel terminal:
watch -n 1 free -h
watch -n 1 'cat /proc/meminfo | grep -E "MemTotal|MemAvailable|MemFree"'

File I/O Benchmarking

Prepare I/O Test Files

# Create test files (required before I/O tests)
sysbench fileio --file-total-size=10G prepare

# Parameters:
# --file-total-size: Total data to write
# --file-num: Number of test files (default 128)

# Verify files created
ls -lah /tmp/sbtest*/
du -sh /tmp/sbtest*/

# Create on specific storage location
sysbench fileio --file-total-size=100G --file-extra-flags=direct \
  --file-test-mode=prepare /mnt/fast-storage/

# View file I/O configuration
sysbench fileio --help | grep file-

Sequential I/O Testing

# Sequential read test
sysbench fileio --file-total-size=10G --file-test-mode=seqrd --time=300 run

# Sequential write test
sysbench fileio --file-total-size=10G --file-test-mode=seqwr --time=300 run

# Sequential rewrite (update existing)
sysbench fileio --file-total-size=10G --file-test-mode=seqrewr --time=300 run

# Measure sustained throughput
sysbench fileio --file-total-size=100G --file-test-mode=seqread \
  --file-extra-flags=direct --time=600 run

Random I/O Testing

# Random read operations
sysbench fileio --file-total-size=10G --file-test-mode=rndrd --time=300 run

# Random write operations
sysbench fileio --file-total-size=10G --file-test-mode=rndwr --time=300 run

# Random read-write mix
sysbench fileio --file-total-size=10G --file-test-mode=rndrw \
  --file-rw-ratio=1:1 --time=300 run

# IOPS-focused short burst test
sysbench fileio --file-total-size=10G --file-test-mode=rndrd \
  --file-io-mode=async --time=60 run

I/O Performance Comparison

# Direct vs buffered I/O
echo "=== Buffered I/O ==="
sysbench fileio --file-total-size=10G --file-test-mode=seqrd --time=60 run

echo "=== Direct I/O (Async) ==="
sysbench fileio --file-total-size=10G --file-test-mode=seqrd \
  --file-extra-flags=direct --file-io-mode=async --time=60 run

# Different queue depths
for depth in 1 4 16 64; do
  echo "=== Queue Depth: $depth ==="
  sysbench fileio --file-total-size=10G --file-test-mode=rndrd \
    --file-io-mode=async --threads=$depth --time=60 run
done

Thread Contention Testing

Mutex Contention Testing

# Test mutex lock contention
sysbench mutex --threads=1 --time=10 run

# Multi-threaded lock contention
sysbench mutex --threads=16 --time=10 run

# Extreme contention test
sysbench mutex --threads=64 --time=10 run

# Monitor lock wait times
# In parallel: watch -n 1 'cat /proc/lock_stat'

Thread Performance Scaling

# Test scaling from 1 to max cores
CORES=$(nproc)

for threads in 1 2 4 8 16 32; do
  if [ $threads -le $CORES ]; then
    echo "=== $threads Threads ==="
    sysbench cpu --threads=$threads --time=10 --cpu-max-prime=20000 run
  fi
done

# Visualize results
cat > analyze_threads.sh <<'EOF'
#!/bin/bash
for threads in 1 2 4 8 16; do
  result=$(sysbench cpu --threads=$threads --time=10 --cpu-max-prime=20000 run 2>&1 | grep "total time:")
  echo "Threads: $threads, $result"
done
EOF

chmod +x analyze_threads.sh
./analyze_threads.sh

Database Benchmarking

OLTP Benchmark Setup

# Prepare OLTP test data
sysbench oltp_prepare --mysql-db=sbtest --mysql-user=root --mysql-password=password

# Or PostgreSQL
sysbench oltp_prepare --pgsql-db=sbtest --pgsql-user=postgres --pgsql-password=password

# Configure benchmark parameters
# --tables: Number of test tables (default 1)
# --table-size: Rows per table (default 10000)
sysbench oltp_prepare --tables=4 --table-size=100000 \
  --mysql-db=sbtest --mysql-user=root --mysql-password=password

Running OLTP Tests

# Transaction throughput test (read-heavy)
sysbench oltp_read_only --threads=8 --time=300 \
  --mysql-db=sbtest --mysql-user=root --mysql-password=password run

# Read-write mixed workload
sysbench oltp_read_write --threads=8 --time=300 \
  --mysql-db=sbtest --mysql-user=root --mysql-password=password run

# Write-heavy workload
sysbench oltp_write_only --threads=8 --time=300 \
  --mysql-db=sbtest --mysql-user=root --mysql-password=password run

# Update-heavy workload
sysbench oltp_update_index --threads=8 --time=300 \
  --mysql-db=sbtest --mysql-user=root --mysql-password=password run

Database Stress Testing

# High concurrency stress test
sysbench oltp_read_write --threads=64 --time=600 \
  --mysql-db=sbtest --mysql-user=root --mysql-password=password run

# Monitor database during test
# In parallel terminal:
mysql -u root -p

# Check from command line
mysqladmin -u root -p status
mysqladmin -u root -p processlist

# View slow query logs
tail -f /var/log/mysql/slow.log

Results Interpretation

Key Performance Metrics

# CPU benchmark results:
# - Total time: Actual elapsed time
# - Total events: Number of calculations
# - Throughput: Operations per second (higher is better)

# Memory benchmark results:
# - Total operations: Data moved
# - Total MiB transferred: Memory bandwidth
# - MiB/sec throughput: Effective memory bandwidth

# I/O benchmark results:
# - Total time: Test duration
# - Total number of events: I/O operations
# - Read MiB/sec: Sequential read bandwidth
# - Write MiB/sec: Sequential write bandwidth
# - total throughput: Total I/Os per second

Sample Output Analysis

# Typical CPU benchmark output:
# CPU speed:
#   events per second:   1234.56   (5.67x from initial estimate)
# General statistics:
#   total time:                          10.0123s
#   total number of events:              12345
# Latency (ms):
#      min:                                0.00
#      avg:                                0.81
#      max:                               23.45
#      95th percentile:                    2.12

# Interpretation:
# Higher events/sec = better CPU performance
# Lower latency = more consistent performance
# Check 95th percentile for worst-case latency

Performance Baseline Comparison

Creating Baseline Results

# Run comprehensive baseline suite
cat > baseline_suite.sh <<'EOF'
#!/bin/bash
DATE=$(date +%Y%m%d_%H%M%S)
RESULTS_DIR="/tmp/sysbench-results-$DATE"
mkdir -p $RESULTS_DIR

# CPU baseline
echo "=== CPU Baseline ==="
sysbench cpu --threads=$(nproc) --time=30 --cpu-max-prime=20000 run | tee $RESULTS_DIR/cpu.txt

# Memory baseline
echo "=== Memory Baseline ==="
sysbench memory --threads=8 --memory-total-size=20G run | tee $RESULTS_DIR/memory.txt

# I/O baseline (prepare first)
echo "=== I/O Baseline (Preparing) ==="
sysbench fileio --file-total-size=10G prepare
echo "=== I/O Baseline (Running) ==="
sysbench fileio --file-total-size=10G --file-test-mode=rndrw --time=60 run | tee $RESULTS_DIR/io.txt

echo "Results saved to $RESULTS_DIR"
EOF

chmod +x baseline_suite.sh
./baseline_suite.sh

Comparing Results

# Extract key metrics for comparison
cat > compare_results.sh <<'EOF'
#!/bin/bash

echo "=== Current Results ==="
sysbench cpu --threads=$(nproc) --time=10 --cpu-max-prime=20000 run | grep -E "total time|events per second|throughput"

# Compare against baseline (stored values)
BASELINE_THROUGHPUT=5000.00

echo ""
echo "=== vs Baseline ==="
CURRENT=$(sysbench cpu --threads=$(nproc) --time=10 --cpu-max-prime=20000 run 2>&1 | grep "events per second" | awk '{print $NF}')
PERCENT=$(echo "scale=2; ($CURRENT / $BASELINE_THROUGHPUT) * 100" | bc)
echo "Baseline throughput: $BASELINE_THROUGHPUT"
echo "Current throughput: $CURRENT"
echo "Percentage change: $PERCENT%"
EOF

chmod +x compare_results.sh
./compare_results.sh

Conclusion

Sysbench provides essential capabilities for infrastructure validation, capacity planning, and performance troubleshooting. By running comprehensive benchmarks across CPU, memory, and I/O subsystems, organizations establish performance baselines enabling objective comparison and optimization. Understanding metric interpretation ensures data-driven decision-making for infrastructure upgrades. Regular benchmarking during infrastructure lifecycle validates performance claims and detects degradation from hardware aging or configuration drift, making sysbench an invaluable tool for modern data center operations.