Disk Space and Inode Usage Monitoring: Complete System Administrator's Guide
Proactive monitoring of disk space and inode usage is fundamental to maintaining healthy Linux systems. Running out of disk space or exhausting inodes can cause system failures, application crashes, and data loss. Understanding how to monitor, analyze, and automate disk usage monitoring prevents these critical issues before they impact production systems.
This comprehensive guide covers all aspects of disk space and inode monitoring, from basic commands to advanced automated monitoring solutions with alerting.
Introduction to Disk Space and Inode Monitoring
Disk usage monitoring involves tracking two critical resources:
- Disk Space: The amount of storage consumed by files and directories
- Inodes: The number of filesystem objects (files, directories, links)
Both resources are finite and can be exhausted independently. A filesystem can run out of space or inodes, either condition preventing new file creation.
Why Monitor Disk Usage?
- Prevent system failures: Full disks cause application crashes and system instability
- Avoid data loss: Insufficient space prevents logging and data writes
- Maintain performance: Full filesystems degrade performance
- Capacity planning: Usage trends inform infrastructure decisions
- Cost optimization: Identify storage waste in cloud environments
- Compliance: Meet retention and storage policies
- Troubleshooting: Quickly identify space-consuming processes
Common Disk Space Issues
- Log files: Unrotated logs filling filesystems
- Temporary files: Accumulated temp files not cleaned up
- Database growth: Expanding databases without monitoring
- User data: Uncontrolled user file uploads
- Backup files: Old backups not deleted
- Core dumps: Large core dumps from crashed processes
- Inode exhaustion: Millions of small files consuming all inodes
Prerequisites
To effectively monitor disk usage, you need:
- Basic understanding of Linux filesystem concepts
- Access to the system (user or root depending on monitored locations)
- Knowledge of filesystem types in use
- Understanding of normal usage patterns
- Familiarity with cron for automation
- Basic scripting knowledge for custom monitoring
Basic Disk Space Monitoring Commands
Using df: Disk Free Space
The df command shows filesystem disk space usage.
Basic Usage
df -h
Output:
Filesystem Size Used Avail Use% Mounted on
/dev/sda1 50G 35G 13G 74% /
/dev/sda2 100G 60G 35G 63% /home
/dev/sdb1 500G 450G 26G 95% /mnt/data
tmpfs 7.8G 1.2M 7.8G 1% /dev/shm
Common df Options
# Human-readable output
df -h
# Include filesystem type
df -hT
# Show inodes instead of blocks
df -hi
# Include all filesystems (including pseudo-filesystems)
df -ha
# Exclude specific filesystem types
df -h -x tmpfs -x devtmpfs
# Show specific filesystem
df -h /home
Understanding df Output
- Filesystem: Device or virtual filesystem
- Size: Total capacity
- Used: Space consumed
- Avail: Available space
- Use%: Percentage used
- Mounted on: Mount point
Using du: Disk Usage
The du command shows disk usage of files and directories.
Basic Usage
# Show directory size
du -sh /var/log
# Show all subdirectories
du -h /var/log
# Limit depth
du -h --max-depth=1 /var
Find Large Directories
# Top 10 largest directories in /var
du -h /var | sort -rh | head -10
# Show sizes for all immediate subdirectories
du -h --max-depth=1 /var | sort -rh
# Find directories larger than 1GB
du -h /var | awk '$1 ~ /^[0-9.]+G/ {print $0}'
Find Large Files
# Find files larger than 100MB in /var
find /var -type f -size +100M -exec du -h {} + | sort -rh
# Top 20 largest files in home directories
find /home -type f -exec du -h {} + | sort -rh | head -20
# Find files larger than 1GB anywhere
sudo find / -type f -size +1G -exec ls -lh {} \; 2>/dev/null
Analyze Specific Directory
# Detailed analysis of /var/log
du -h /var/log | sort -rh | head -20
# With file count
du -sh /var/log/* | sort -rh
Inode Monitoring
Understanding Inodes
Each file and directory consumes one inode. Running out of inodes prevents file creation even with available disk space.
Check Inode Usage
# Show inode usage
df -hi
# Show inode usage for specific filesystem
df -hi /home
Output:
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/sda1 3.2M 500K 2.7M 16% /
/dev/sda2 6.5M 2.5M 4.0M 39% /home
/dev/sdb1 32M 25M 6.8M 79% /mnt/data
Find Directories with Many Files
# Count files in directories
for dir in /var/*; do
echo -n "$dir: "
find "$dir" -type f | wc -l
done
# Find directories with most files
find /var -type d -exec sh -c 'echo -n "{}: "; find "{}" -maxdepth 1 -type f | wc -l' \; | sort -t: -k2 -rn | head -10
# Count files in immediate subdirectories
for dir in /var/log/*; do
count=$(find "$dir" -type f 2>/dev/null | wc -l)
echo "$count $dir"
done | sort -rn
Find File Systems with Many Small Files
# Average file size per filesystem
df -h | awk 'NR>1 {print $6}' | while read mount; do
total_size=$(du -sb "$mount" 2>/dev/null | awk '{print $1}')
file_count=$(find "$mount" -type f 2>/dev/null | wc -l)
if [ "$file_count" -gt 0 ]; then
avg_size=$((total_size / file_count))
echo "$mount: $file_count files, avg size: $(( avg_size / 1024 ))KB"
fi
done
Real-Time Monitoring
Watch Disk Space in Real-Time
# Update every 2 seconds
watch -n 2 'df -h'
# Monitor specific filesystem
watch -n 2 'df -h /var/log'
# Monitor with inode information
watch -n 2 'df -h && echo && df -hi'
Monitor Disk Usage Changes
# Monitor directory size changes
watch -n 5 'du -sh /var/log'
# Track fastest growing directory
watch -n 10 'du -h --max-depth=1 /var | sort -rh | head -5'
Monitor File Creation Rate
# Count files created in last minute
watch -n 60 'find /tmp -type f -mmin -1 | wc -l'
Automated Monitoring Scripts
Basic Disk Space Monitoring Script
sudo nano /usr/local/bin/disk_monitor.sh
#!/bin/bash
# Disk Space Monitoring Script
THRESHOLD=80
EMAIL="[email protected]"
LOGFILE="/var/log/disk_monitor.log"
# Check each filesystem
df -H | grep -vE '^Filesystem|tmpfs|cdrom' | awk '{ print $5 " " $6 }' | while read output;
do
usage=$(echo $output | awk '{print $1}' | sed 's/%//g')
partition=$(echo $output | awk '{print $2}')
if [ $usage -ge $THRESHOLD ]; then
timestamp=$(date '+%Y-%m-%d %H:%M:%S')
message="$timestamp - WARNING: Disk usage on $partition is at ${usage}%"
echo "$message" >> $LOGFILE
# Send email alert (requires mail command)
echo "$message" | mail -s "Disk Space Alert: $partition" $EMAIL
fi
done
Make executable:
sudo chmod +x /usr/local/bin/disk_monitor.sh
Schedule with cron:
# Run every hour
echo "0 * * * * /usr/local/bin/disk_monitor.sh" | sudo crontab -
Advanced Monitoring Script with Inode Check
sudo nano /usr/local/bin/disk_inode_monitor.sh
#!/bin/bash
# Disk Space and Inode Monitoring Script
DISK_THRESHOLD=80
INODE_THRESHOLD=80
EMAIL="[email protected]"
LOGFILE="/var/log/disk_inode_monitor.log"
timestamp=$(date '+%Y-%m-%d %H:%M:%S')
# Check disk space
echo "[$timestamp] Disk Space Check" >> $LOGFILE
df -H | grep -vE '^Filesystem|tmpfs|cdrom' | awk '{ print $5 " " $6 }' | while read output;
do
usage=$(echo $output | awk '{print $1}' | sed 's/%//g')
partition=$(echo $output | awk '{print $2}')
if [ $usage -ge $DISK_THRESHOLD ]; then
alert="WARNING: Disk usage on $partition is at ${usage}%"
echo "$alert" >> $LOGFILE
echo "$alert" | mail -s "Disk Space Alert: $partition" $EMAIL
fi
done
# Check inode usage
echo "[$timestamp] Inode Usage Check" >> $LOGFILE
df -i | grep -vE '^Filesystem|tmpfs|cdrom' | awk '{ print $5 " " $6 }' | while read output;
do
usage=$(echo $output | awk '{print $1}' | sed 's/%//g')
partition=$(echo $output | awk '{print $2}')
if [ $usage -ge $INODE_THRESHOLD ]; then
alert="WARNING: Inode usage on $partition is at ${usage}%"
echo "$alert" >> $LOGFILE
echo "$alert" | mail -s "Inode Alert: $partition" $EMAIL
fi
done
Growth Tracking Script
sudo nano /usr/local/bin/disk_growth_tracker.sh
#!/bin/bash
# Disk Growth Tracking Script
DATAFILE="/var/log/disk_growth.log"
REPORTFILE="/var/log/disk_growth_report.txt"
# Record current usage
timestamp=$(date '+%Y-%m-%d %H:%M:%S')
df -BG | grep -vE '^Filesystem|tmpfs' | awk -v ts="$timestamp" '{print ts "," $6 "," $3}' >> $DATAFILE
# Generate weekly growth report (run on Sundays)
if [ $(date +%u) -eq 7 ]; then
{
echo "Weekly Disk Growth Report"
echo "Generated: $(date)"
echo "================================"
echo ""
# Calculate growth for each filesystem
for mount in $(df | grep -vE '^Filesystem|tmpfs' | awk '{print $6}'); do
echo "Filesystem: $mount"
# Get data from last 7 days
week_ago=$(date -d '7 days ago' '+%Y-%m-%d')
current=$(grep "$mount" $DATAFILE | tail -1 | cut -d',' -f3)
previous=$(grep "$mount" $DATAFILE | grep "$week_ago" | tail -1 | cut -d',' -f3)
if [ -n "$current" ] && [ -n "$previous" ]; then
growth=$((${current%G} - ${previous%G}))
echo " Growth: ${growth}GB in 7 days"
fi
echo ""
done
} > $REPORTFILE
# Email report
mail -s "Weekly Disk Growth Report" [email protected] < $REPORTFILE
fi
Directory Size Monitoring Script
sudo nano /usr/local/bin/directory_size_monitor.sh
#!/bin/bash
# Monitor specific directories for size changes
MONITORED_DIRS="/var/log /home /var/lib/mysql"
THRESHOLD_GB=10
LOGFILE="/var/log/directory_size_monitor.log"
for dir in $MONITORED_DIRS; do
if [ -d "$dir" ]; then
size=$(du -sb "$dir" 2>/dev/null | awk '{print $1}')
size_gb=$((size / 1024 / 1024 / 1024))
timestamp=$(date '+%Y-%m-%d %H:%M:%S')
echo "$timestamp - $dir: ${size_gb}GB" >> $LOGFILE
if [ $size_gb -ge $THRESHOLD_GB ]; then
message="$dir has reached ${size_gb}GB"
echo "$message" | mail -s "Directory Size Alert" [email protected]
fi
fi
done
Disk Space Analysis Tools
Using ncdu (NCurses Disk Usage)
Interactive disk usage analyzer.
Install:
# Debian/Ubuntu
sudo apt install ncdu
# CentOS/RHEL
sudo yum install ncdu
Usage:
# Analyze directory
ncdu /var
# Analyze with one filesystem
ncdu -x /var
# Export to file
ncdu -o /tmp/var_usage.txt /var
# Import from file
ncdu -f /tmp/var_usage.txt
Using agedu (Age-Based Disk Usage)
Find old, large files.
Install:
sudo apt install agedu
Usage:
# Index directory
agedu -s /home
# Generate report of files unused in 6 months
agedu -a /home -t 180d
# Web interface
agedu -w --address 0.0.0.0:8000
# Access at http://localhost:8000
Using baobab (GUI Tool)
Graphical disk usage analyzer.
Install:
sudo apt install baobab
Usage:
# Launch GUI
baobab /var
Cleaning Up Disk Space
Safe Cleanup Procedures
Clean Package Manager Cache
# Debian/Ubuntu
sudo apt clean
sudo apt autoclean
sudo apt autoremove
# CentOS/RHEL
sudo yum clean all
sudo dnf clean all
Clean Journald Logs
# Keep only last 7 days
sudo journalctl --vacuum-time=7d
# Keep only 500MB
sudo journalctl --vacuum-size=500M
# Verify space freed
du -sh /var/log/journal
Rotate and Compress Logs
# Force log rotation
sudo logrotate -f /etc/logrotate.conf
# Clean old compressed logs
sudo find /var/log -type f -name "*.gz" -mtime +30 -delete
sudo find /var/log -type f -name "*.old" -mtime +30 -delete
Clean Temporary Files
# Clean /tmp (files older than 10 days)
sudo find /tmp -type f -atime +10 -delete
# Clean /var/tmp
sudo find /var/tmp -type f -atime +30 -delete
# Clean systemd temporary files
sudo systemctl clean --what=cache --all
Clean User Caches
# Clean user cache directories
find ~/.cache -type f -atime +30 -delete
# Clean browser caches (when logged in as user)
rm -rf ~/.cache/mozilla
rm -rf ~/.cache/google-chrome
rm -rf ~/.cache/chromium
Find and Remove Core Dumps
# Find core dumps
sudo find / -name "core.*" -type f 2>/dev/null
# Remove core dumps
sudo find / -name "core.*" -type f -delete 2>/dev/null
# Disable core dumps
echo "* hard core 0" | sudo tee -a /etc/security/limits.conf
Automated Cleanup Script
sudo nano /usr/local/bin/disk_cleanup.sh
#!/bin/bash
# Automated Disk Cleanup Script
LOGFILE="/var/log/disk_cleanup.log"
timestamp=$(date '+%Y-%m-%d %H:%M:%S')
echo "[$timestamp] Starting cleanup" >> $LOGFILE
# Package cache
echo "Cleaning package cache..." >> $LOGFILE
apt clean 2>&1 | tee -a $LOGFILE
# Old kernels (Debian/Ubuntu)
echo "Removing old kernels..." >> $LOGFILE
apt autoremove -y 2>&1 | tee -a $LOGFILE
# Journal logs (keep 7 days)
echo "Cleaning journal logs..." >> $LOGFILE
journalctl --vacuum-time=7d 2>&1 | tee -a $LOGFILE
# Temporary files
echo "Cleaning temporary files..." >> $LOGFILE
find /tmp -type f -atime +7 -delete 2>&1 | tee -a $LOGFILE
find /var/tmp -type f -atime +30 -delete 2>&1 | tee -a $LOGFILE
# Rotate logs
echo "Rotating logs..." >> $LOGFILE
logrotate -f /etc/logrotate.conf 2>&1 | tee -a $LOGFILE
# Old compressed logs
echo "Removing old compressed logs..." >> $LOGFILE
find /var/log -name "*.gz" -mtime +60 -delete 2>&1 | tee -a $LOGFILE
# Report space freed
echo "[$timestamp] Cleanup completed" >> $LOGFILE
df -h >> $LOGFILE
# Email report
mail -s "Disk Cleanup Report" [email protected] < $LOGFILE
Schedule weekly:
echo "0 2 * * 0 /usr/local/bin/disk_cleanup.sh" | sudo crontab -
Monitoring Dashboard Creation
Create Daily Disk Report
sudo nano /usr/local/bin/daily_disk_report.sh
#!/bin/bash
# Daily Disk Usage Report
REPORT_FILE="/var/log/daily_disk_report_$(date +%Y%m%d).txt"
{
echo "DISK USAGE REPORT"
echo "Generated: $(date)"
echo "Server: $(hostname)"
echo "========================================"
echo ""
echo "FILESYSTEM USAGE:"
echo "----------------"
df -h | grep -vE 'tmpfs|cdrom'
echo ""
echo "INODE USAGE:"
echo "------------"
df -hi | grep -vE 'tmpfs|cdrom'
echo ""
echo "TOP 10 LARGEST DIRECTORIES:"
echo "---------------------------"
du -h --max-depth=2 / 2>/dev/null | sort -rh | head -10
echo ""
echo "TOP 10 LARGEST FILES:"
echo "---------------------"
find / -type f -exec du -h {} + 2>/dev/null | sort -rh | head -10
echo ""
echo "DIRECTORIES WITH MOST FILES:"
echo "----------------------------"
for dir in /var /home /opt; do
echo -n "$dir: "
find "$dir" -type f 2>/dev/null | wc -l
done
echo ""
} > $REPORT_FILE
# Email report
mail -s "Daily Disk Usage Report - $(hostname)" [email protected] < $REPORT_FILE
# Keep only last 30 days of reports
find /var/log -name "daily_disk_report_*.txt" -mtime +30 -delete
Troubleshooting Disk Space Issues
Identify What's Using Space When df and du Disagree
# Check for deleted files still held open
sudo lsof | grep deleted
# Find process holding deleted file
sudo lsof +L1
# Restart service to release file handles
sudo systemctl restart service_name
Find Hidden Space Usage
# Check all mount points
df -h
# Check for files in mount point directories
for mount in $(df | grep -vE 'tmpfs|cdrom' | awk '{print $6}'); do
echo "Checking: $mount"
sudo umount "$mount" 2>/dev/null && \
du -sh "$mount" && \
sudo mount "$mount"
done
Investigate Sudden Space Loss
# Find recently created large files
sudo find / -type f -size +100M -mtime -1 -exec ls -lh {} \; 2>/dev/null
# Find files modified in last hour
sudo find / -type f -mmin -60 -size +10M 2>/dev/null
# Check system logs for clues
sudo journalctl --since "1 hour ago" | grep -i error
Best Practices
1. Set Up Proactive Monitoring
Don't wait for disk full alerts:
# Monitor at 80% threshold, not 95%
THRESHOLD=80
2. Monitor Growth Trends
Track usage over time:
# Daily snapshots for trend analysis
echo "0 0 * * * df -h >> /var/log/disk_usage_$(date +\%Y\%m).log" | crontab -
3. Implement Log Rotation
Configure logrotate properly:
sudo nano /etc/logrotate.d/custom_app
/var/log/custom_app/*.log {
daily
rotate 7
compress
delaycompress
missingok
notifempty
}
4. Regular Cleanup Schedules
Automate cleanup tasks:
# Weekly cleanup every Sunday at 2 AM
0 2 * * 0 /usr/local/bin/disk_cleanup.sh
5. Use Separate Partitions
Isolate high-growth areas:
- Separate /var/log partition
- Separate /home partition
- Separate /tmp partition
- Separate application data partitions
6. Set Up Alerts at Multiple Thresholds
# Warning at 75%, critical at 90%
WARNING_THRESHOLD=75
CRITICAL_THRESHOLD=90
7. Document Normal Usage Patterns
Know your baseline:
# Monthly usage report
df -h > /root/disk_usage_baseline_$(date +%Y%m).txt
8. Monitor Inode Usage
Don't forget inodes:
# Include inode checks in monitoring
df -hi
9. Keep Historical Data
Maintain usage history:
# Archive monthly reports
mkdir -p /var/log/disk_history
mv /var/log/disk_usage_*.log /var/log/disk_history/
10. Test Cleanup Procedures
Verify cleanup scripts work:
# Test in non-production first
# Verify expected files are deleted
# Confirm space is actually freed
Conclusion
Effective disk space and inode monitoring is critical for maintaining healthy Linux systems. By implementing proactive monitoring, automated alerting, regular cleanup procedures, and trend analysis, you prevent disk-full emergencies and maintain optimal system performance.
Key takeaways:
- Monitor both space and inodes - either can be exhausted
- Set alert thresholds at 75-80%, not 95%
- Automate monitoring with scripts and cron jobs
- Track growth trends over time for capacity planning
- Implement log rotation to prevent log file growth
- Schedule regular cleanup of temporary files and caches
- Use appropriate tools - df, du, ncdu for different needs
- Document baselines for normal usage patterns
- Test cleanup procedures before emergencies
- Maintain historical data for trend analysis
Remember that disk monitoring isn't just about preventing full disks—it's about understanding system behavior, optimizing storage usage, and planning for future capacity needs. By following the practices and scripts in this guide, you can implement comprehensive disk monitoring that keeps your Linux infrastructure running smoothly.
Proactive monitoring prevents reactive firefighting. Invest time in proper monitoring infrastructure now to avoid critical outages later.