Log Retention and Rotation: Complete Implementation Guide

Introduction

Log retention and rotation are critical components of system administration, security monitoring, compliance, and troubleshooting. Properly managed logs provide an essential audit trail for security investigations, help diagnose system issues, enable performance analysis, and demonstrate compliance with regulatory requirements. However, logs that are not properly rotated can consume massive amounts of disk space, while logs retained for too short a period may not be available when needed for forensic analysis or compliance audits.

This comprehensive guide provides Linux system administrators with practical knowledge for implementing effective log rotation policies, configuring retention periods that meet both operational and compliance requirements, and establishing automated procedures for log management. Whether you're managing a single server or a large-scale infrastructure, this guide covers the essential tools, configurations, and best practices for maintaining comprehensive, compliant, and manageable log systems.

Why Log Retention and Rotation Matter

Operational Benefits:

• Disk Space Management: Prevents log files from consuming all available storage
• Performance Optimization: Keeps log files at manageable sizes for faster searching and processing
• Troubleshooting: Maintains historical data for diagnosing recurring or intermittent issues
• Capacity Planning: Provides data for trend analysis and infrastructure planning

Security and Compliance Benefits:

• Security Incident Investigation: Provides forensic evidence for security breach analysis
• Compliance Requirements: Meets regulatory requirements for log retention (GDPR, PCI-DSS, HIPAA, SOX)
• Audit Trail: Demonstrates system activity and changes for auditors
• Intrusion Detection: Enables detection of attack patterns over time

Regulatory Requirements for Log Retention

Different regulations mandate specific log retention periods:

GDPR (General Data Protection Regulation):

• No specific retention period mandated
• Logs must be retained only as long as necessary for stated purpose
• Personal data in logs subject to data minimization principles
• Typically 6 months to 2 years for security logs

PCI-DSS (Payment Card Industry Data Security Standard):

• Minimum 3 months immediately available
• Minimum 1 year total retention
• System component logs must be retained and protected

HIPAA (Health Insurance Portability and Accountability Act):

• Minimum 6 years for audit logs
• Security incident logs retained for investigation period

SOX (Sarbanes-Oxley Act):

• Minimum 7 years for financial systems
• Access logs, change logs, and audit trails required

ISO 27001:

• Typically requires 12 months of log retention
• Security events retained for analysis and investigation

This guide helps you implement retention policies that meet these requirements while maintaining operational efficiency.

Understanding Log Rotation

What is Log Rotation?

Log rotation is the automated process of:

1. Renaming current log files
2. Creating new log files
3. Compressing old log files
4. Deleting very old log files based on retention policy
5. Notifying services to write to new log files

Rotation Methods

Time-Based Rotation: Rotate logs based on time intervals (daily, weekly, monthly)

• Best for: Predictable log sizes, compliance requirements with time-based retention
• Example: Rotate daily, keep 30 days

Size-Based Rotation: Rotate logs when they reach a specific size

• Best for: High-volume logging, preventing individual files from becoming too large
• Example: Rotate at 100MB, keep 10 files

Combined Approach: Rotate based on both time and size

• Best for: Variable log volumes with size limits
• Example: Rotate daily or at 500MB, whichever comes first

Logrotate - The Standard Tool

Logrotate is the standard log rotation utility on Linux systems, providing flexible and powerful log management capabilities.

Installing Logrotate

# Install logrotate (usually pre-installed)
sudo apt-get update
sudo apt-get install -y logrotate # Debian/Ubuntu
sudo yum install -y logrotate # RHEL/CentOS

# Verify installation
logrotate --version

# Check if logrotate is scheduled
systemctl status logrotate.timer # systemd-based systems
ls -la /etc/cron.daily/logrotate # cron-based systems

Logrotate Configuration Structure

# Main configuration file
cat /etc/logrotate.conf

# Configuration directory for individual services
ls -la /etc/logrotate.d/

# Logrotate state file (tracks rotation status)
cat /var/lib/logrotate/status

Basic Logrotate Configuration

# Create custom logrotate configuration
sudo tee /etc/logrotate.d/custom-app << 'EOF'
# Custom Application Log Rotation

/var/log/custom-app/*.log {
    # Rotate daily
    daily

    # Keep 30 days of logs
    rotate 30

    # Compress old logs
    compress

    # Delay compression by one rotation cycle
    delaycompress

    # Don't rotate if log is empty
    notifempty

    # Create new log file with specific permissions
    create 0640 www-data adm

    # Use date as extension instead of numbers
    dateext

    # Date format
    dateformat -%Y%m%d

    # Post-rotation script
    postrotate
        # Reload service to use new log file
        systemctl reload custom-app > /dev/null 2>&1 || true
    endscript
}
EOF

# Test configuration without actually rotating
sudo logrotate -d /etc/logrotate.d/custom-app

# Force rotation (for testing)
sudo logrotate -f /etc/logrotate.d/custom-app

# Check rotation status
sudo cat /var/lib/logrotate/status | grep custom-app

Logrotate Directives Explained

# Create comprehensive example with all common directives
sudo tee /etc/logrotate.d/comprehensive-example << 'EOF'
# Comprehensive Logrotate Example

/var/log/app/*.log {
    # ROTATION FREQUENCY
    daily          # Rotate daily (also: hourly, weekly, monthly, yearly)

    # RETENTION POLICY
    rotate 90      # Keep 90 rotated logs

    # SIZE CONSTRAINTS
    size 100M      # Rotate if log exceeds 100MB (also: k, M, G)
    maxsize 500M   # Force rotation at 500MB regardless of schedule

    # COMPRESSION
    compress       # Compress rotated logs with gzip
    delaycompress  # Compress on next rotation (keeps previous log uncompressed)
    compresscmd /usr/bin/gzip  # Compression command
    compressext .gz            # Compression extension
    compressoptions -9         # Maximum compression

    # FILE HANDLING
    create 0640 www-data adm   # Create new file with mode, owner, group
    # Or: nocreate               # Don't create new file
    # Or: copytruncate          # Copy then truncate (for apps that can't reopen files)

    # EMPTY FILES
    notifempty     # Don't rotate if log is empty
    # Or: ifempty   # Rotate even if empty

    # MISSING FILES
    missingok      # Don't error if log file is missing
    # Or: nomissingok # Error if log file is missing

    # DATING
    dateext        # Use date in rotated filename
    dateformat -%Y%m%d-%s  # Date format
    dateyesterday  # Use yesterday's date

    # OLD DIRECTORY
    olddir /var/log/app/archive  # Move rotated logs to different directory
    # createolddir 0755 root root  # Create olddir if missing

    # MAIL
    mail [email protected]  # Email log before deletion
    mailfirst      # Mail first rotated log
    # Or: maillast  # Mail last rotated log before deletion

    # SHARED SCRIPTS
    sharedscripts  # Run postrotate script once for all logs

    # EXTENSION
    extension .log # Force specific extension

    # PRE/POST ROTATION SCRIPTS
    prerotate
        # Commands to run before rotation
        /usr/local/bin/pre-rotation-backup.sh
    endscript

    postrotate
        # Commands to run after rotation
        systemctl reload app > /dev/null 2>&1 || true
    endscript

    # LAST ACTION
    lastaction
        # Commands to run after all logs rotated
        /usr/local/bin/cleanup.sh
    endscript
}
EOF

Apache/Nginx Log Rotation

# Apache log rotation configuration
sudo tee /etc/logrotate.d/apache2 << 'EOF'
/var/log/apache2/*.log {
    daily
    rotate 52        # Keep 52 weeks (1 year)
    compress
    delaycompress
    notifempty
    create 0640 root adm
    dateext
    sharedscripts
    postrotate
        # Reload Apache gracefully
        if /etc/init.d/apache2 status > /dev/null ; then \
            /etc/init.d/apache2 reload > /dev/null; \
        fi;
    endscript
}

# Separate configuration for access logs (higher retention for compliance)
/var/log/apache2/access*.log {
    daily
    rotate 365       # Keep 1 year for compliance
    compress
    delaycompress
    notifempty
    create 0640 root adm
    dateext
    sharedscripts
    postrotate
        if /etc/init.d/apache2 status > /dev/null ; then \
            /etc/init.d/apache2 reload > /dev/null; \
        fi;
    endscript
}
EOF

# Nginx log rotation configuration
sudo tee /etc/logrotate.d/nginx << 'EOF'
/var/log/nginx/*.log {
    daily
    rotate 52
    compress
    delaycompress
    notifempty
    create 0640 www-data adm
    dateext
    sharedscripts
    postrotate
        # Send USR1 signal to reopen log files
        [ -f /var/run/nginx.pid ] && kill -USR1 $(cat /var/run/nginx.pid)
    endscript
}
EOF

MySQL/MariaDB Log Rotation

# MySQL log rotation configuration
sudo tee /etc/logrotate.d/mysql-server << 'EOF'
# MySQL Error Log
/var/log/mysql/error.log {
    daily
    rotate 365       # Keep 1 year
    compress
    delaycompress
    notifempty
    create 0640 mysql adm
    dateext
    sharedscripts
    postrotate
        # Flush MySQL logs
        test -x /usr/bin/mysqladmin && \
        /usr/bin/mysqladmin --defaults-file=/etc/mysql/debian.cnf flush-logs
    endscript
}

# MySQL Slow Query Log
/var/log/mysql/slow-query.log {
    daily
    rotate 90        # Keep 90 days
    compress
    delaycompress
    notifempty
    create 0640 mysql adm
    dateext
    maxsize 1G
    sharedscripts
    postrotate
        test -x /usr/bin/mysqladmin && \
        /usr/bin/mysqladmin --defaults-file=/etc/mysql/debian.cnf flush-logs
    endscript
}

# MySQL General Query Log
/var/log/mysql/query.log {
    size 100M        # Rotate at 100MB (can grow quickly)
    rotate 10        # Keep only 10 rotations
    compress
    delaycompress
    notifempty
    create 0640 mysql adm
    dateext
    sharedscripts
    postrotate
        test -x /usr/bin/mysqladmin && \
        /usr/bin/mysqladmin --defaults-file=/etc/mysql/debian.cnf flush-logs
    endscript
}
EOF

PostgreSQL Log Rotation

# PostgreSQL log rotation configuration
sudo tee /etc/logrotate.d/postgresql << 'EOF'
/var/log/postgresql/postgresql-*.log {
    daily
    rotate 365
    compress
    delaycompress
    notifempty
    create 0640 postgres postgres
    dateext
    missingok
    sharedscripts
    postrotate
        # Send SIGHUP to PostgreSQL to reopen log files
        /usr/bin/pg_ctlcluster --skip-systemctl-redirect 13 main reload > /dev/null
    endscript
}
EOF

Syslog Rotation

# Rsyslog rotation configuration
sudo tee /etc/logrotate.d/rsyslog << 'EOF'
/var/log/syslog
/var/log/mail.info
/var/log/mail.warn
/var/log/mail.err
/var/log/daemon.log
/var/log/kern.log
/var/log/auth.log
/var/log/user.log
{
    daily
    rotate 90        # Keep 90 days
    compress
    delaycompress
    notifempty
    create 0640 syslog adm
    dateext
    sharedscripts
    postrotate
        # Tell rsyslog to reopen log files
        /usr/lib/rsyslog/rsyslog-rotate
    endscript
}

# Critical logs with longer retention
/var/log/auth.log
/var/log/secure
{
    daily
    rotate 365       # Keep 1 year for security audit
    compress
    delaycompress
    notifempty
    create 0600 root root
    dateext
    sharedscripts
    postrotate
        /usr/lib/rsyslog/rsyslog-rotate
    endscript
}
EOF

Compliance-Driven Retention Policies

PCI-DSS Compliant Log Rotation

# PCI-DSS requires 3 months immediately available, 1 year total
sudo tee /etc/logrotate.d/pci-compliance << 'EOF'
# PCI-DSS Audit Logs (Payment Processing Systems)

# Application logs
/var/log/payment-app/*.log {
    daily
    rotate 365       # 1 year total retention
    compress
    delaycompress
    notifempty
    create 0600 root root
    dateext
    dateformat -%Y%m%d

    # First 90 days uncompressed (immediately available)
    # This is handled by delaycompress and manual processes

    sharedscripts
    postrotate
        systemctl reload payment-app > /dev/null 2>&1 || true
    endscript
}

# Access logs (cardholder data environment)
/var/log/nginx/payment*.log {
    daily
    rotate 365
    compress
    delaycompress
    notifempty
    create 0640 www-data adm
    dateext
    sharedscripts
    postrotate
        [ -f /var/run/nginx.pid ] && kill -USR1 $(cat /var/run/nginx.pid)
    endscript
}

# Database audit logs
/var/log/mysql/audit.log {
    daily
    rotate 365
    compress
    delaycompress
    notifempty
    create 0600 mysql mysql
    dateext
    maxsize 1G
    sharedscripts
    postrotate
        mysqladmin --defaults-file=/etc/mysql/debian.cnf flush-logs
    endscript
}

# System audit logs (auditd)
/var/log/audit/audit.log {
    daily
    rotate 365
    compress
    delaycompress
    notifempty
    create 0600 root root
    dateext
    sharedscripts
    postrotate
        /sbin/service auditd rotate > /dev/null 2>&1 || true
    endscript
}
EOF

GDPR Compliant Log Rotation

# GDPR-compliant log rotation with privacy considerations
sudo tee /etc/logrotate.d/gdpr-compliance << 'EOF'
# GDPR Compliant Log Rotation

# Web server logs (contain personal data - IP addresses)
/var/log/nginx/access.log
/var/log/apache2/access.log {
    daily
    rotate 180       # 6 months retention (adjust per your privacy policy)
    compress
    delaycompress
    notifempty
    create 0640 www-data adm
    dateext

    sharedscripts

    # Anonymize IP addresses before compression
    prerotate
        # Anonymize last rotated log
        if [ -f /var/log/nginx/access.log.1 ]; then
            /usr/local/bin/anonymize-logs.sh /var/log/nginx/access.log.1
        fi
    endscript

    postrotate
        [ -f /var/run/nginx.pid ] && kill -USR1 $(cat /var/run/nginx.pid)
        /etc/init.d/apache2 reload > /dev/null 2>&1 || true
    endscript
}

# Application logs (minimal personal data retention)
/var/log/app/user-activity.log {
    daily
    rotate 90        # 90 days only
    compress
    delaycompress
    notifempty
    create 0640 app-user adm
    dateext

    # Sanitize personal data before rotation
    prerotate
        /usr/local/bin/gdpr-sanitize.sh /var/log/app/user-activity.log.1
    endscript

    postrotate
        systemctl reload app-service > /dev/null 2>&1 || true
    endscript
}

# Security logs (legitimate interest, longer retention)
/var/log/auth.log
/var/log/security.log {
    daily
    rotate 730       # 2 years for security incident investigation
    compress
    delaycompress
    notifempty
    create 0600 root root
    dateext

    postrotate
        /usr/lib/rsyslog/rsyslog-rotate
    endscript
}
EOF

# Create log anonymization script
sudo tee /usr/local/bin/anonymize-logs.sh << 'EOF'
#!/bin/bash
# Anonymize IP addresses in log files for GDPR compliance

LOG_FILE="$1"

if [ ! -f "$LOG_FILE" ]; then
    exit 0
fi

# Create temporary file
TEMP_FILE=$(mktemp)

# Anonymize IPv4 addresses (mask last octet)
# 192.168.1.100 becomes 192.168.1.0
sed -E 's/([0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.)[0-9]{1,3}/\10/g' "$LOG_FILE" > "$TEMP_FILE"

# Anonymize IPv6 addresses (mask last 4 groups)
# 2001:db8::1 becomes 2001:db8::0
sed -i -E 's/([0-9a-f]{1,4}::[0-9a-f]{1,4})/\1:0/g' "$TEMP_FILE"

# Replace original file
mv "$TEMP_FILE" "$LOG_FILE"
chmod 640 "$LOG_FILE"

EOF

sudo chmod +x /usr/local/bin/anonymize-logs.sh

HIPAA Compliant Log Rotation

# HIPAA requires 6 years retention for audit logs
sudo tee /etc/logrotate.d/hipaa-compliance << 'EOF'
# HIPAA Compliant Log Rotation (Healthcare Systems)

# Application audit logs
/var/log/healthcare-app/audit.log {
    daily
    rotate 2190      # 6 years retention (365 * 6)
    compress
    delaycompress
    notifempty
    create 0600 root root
    dateext
    dateformat -%Y%m%d

    # After 90 days, move to archive storage
    lastaction
        find /var/log/healthcare-app -name "audit.log-*.gz" -mtime +90 \
            -exec mv {} /archive/logs/healthcare-app/ \;
    endscript

    postrotate
        systemctl reload healthcare-app > /dev/null 2>&1 || true
    endscript
}

# Database access logs
/var/log/mysql/access-audit.log {
    daily
    rotate 2190      # 6 years
    compress
    delaycompress
    notifempty
    create 0600 mysql mysql
    dateext
    maxsize 500M

    postrotate
        mysqladmin --defaults-file=/etc/mysql/debian.cnf flush-logs
    endscript
}

# System access logs
/var/log/auth.log
/var/log/secure {
    daily
    rotate 2190      # 6 years
    compress
    delaycompress
    notifempty
    create 0600 root root
    dateext

    postrotate
        /usr/lib/rsyslog/rsyslog-rotate
    endscript
}
EOF

Advanced Log Management

Size-Based Rotation with Limits

# High-volume application with size limits
sudo tee /etc/logrotate.d/high-volume-app << 'EOF'
/var/log/high-volume-app/*.log {
    # Rotate when log reaches 500MB
    size 500M

    # Keep last 20 rotations
    rotate 20

    # But never keep more than 30 days
    maxage 30

    compress
    delaycompress
    notifempty
    create 0640 app-user adm
    dateext

    # If log grows beyond 2GB, force immediate rotation
    maxsize 2G

    postrotate
        systemctl reload high-volume-app > /dev/null 2>&1 || true
    endscript
}
EOF

Hourly Log Rotation

# For extremely high-volume logs
sudo tee /etc/logrotate.d/hourly-rotation << 'EOF'
/var/log/streaming-app/*.log {
    hourly           # Rotate every hour
    rotate 168       # Keep 1 week (24 * 7)
    compress
    delaycompress
    notifempty
    create 0640 stream-user adm
    dateext
    dateformat -%Y%m%d-%H

    postrotate
        systemctl reload streaming-app > /dev/null 2>&1 || true
    endscript
}
EOF

# Create hourly cron job for logrotate
sudo tee /etc/cron.hourly/logrotate-hourly << 'EOF'
#!/bin/bash
/usr/sbin/logrotate /etc/logrotate.d/hourly-rotation
EOF

sudo chmod +x /etc/cron.hourly/logrotate-hourly

Copytruncate Method

# For applications that can't reopen log files
sudo tee /etc/logrotate.d/copytruncate-example << 'EOF'
/var/log/legacy-app/application.log {
    daily
    rotate 30
    compress

    # Copy and truncate instead of move and create
    copytruncate

    notifempty
    dateext
}
EOF

Multi-Level Archiving

# Recent logs on fast storage, old logs on archive storage
sudo tee /etc/logrotate.d/multi-level-archive << 'EOF'
# Recent logs (last 30 days on SSD)
/var/log/application/*.log {
    daily
    rotate 30
    compress
    delaycompress
    notifempty
    create 0640 app-user adm
    dateext

    # Move logs older than 30 days to archive storage
    lastaction
        find /var/log/application -name "*.log-*.gz" -mtime +30 \
            -exec mv {} /mnt/archive-storage/logs/application/ \;
    endscript

    postrotate
        systemctl reload application > /dev/null 2>&1 || true
    endscript
}
EOF

# Additional script to handle archive storage rotation
sudo tee /etc/logrotate.d/archive-storage << 'EOF'
# Archive storage (keep 1 year)
/mnt/archive-storage/logs/application/*.gz {
    monthly
    rotate 12
    compress
    notifempty
    maxage 365
}
EOF

Centralized Logging with Rotation

Rsyslog Central Server

# Configure rsyslog to receive remote logs
sudo tee /etc/rsyslog.d/remote-logs.conf << 'EOF'
# Enable UDP syslog reception
module(load="imudp")
input(type="imudp" port="514")

# Enable TCP syslog reception
module(load="imtcp")
input(type="imtcp" port="514")

# Template for remote host logs
template(name="RemoteHost" type="string"
         string="/var/log/remote/%HOSTNAME%/%PROGRAMNAME%.log")

# Log remote hosts to separate files
if $fromhost-ip != '127.0.0.1' then ?RemoteHost
& stop
EOF

sudo systemctl restart rsyslog

# Configure rotation for remote host logs
sudo tee /etc/logrotate.d/remote-logs << 'EOF'
/var/log/remote/*/*.log {
    daily
    rotate 90
    compress
    delaycompress
    notifempty
    create 0640 syslog adm
    dateext

    # Organize by hostname and date
    olddir archive
    createolddir 0755 syslog adm

    sharedscripts
    postrotate
        /usr/lib/rsyslog/rsyslog-rotate
    endscript
}
EOF

Syslog Client Configuration

# Configure client to send logs to central server
sudo tee /etc/rsyslog.d/central-logging.conf << 'EOF'
# Send all logs to central server
*.* @@central-log-server:514  # TCP
*.* @central-log-server:514   # UDP (less reliable but faster)

# Also keep local copies
*.* /var/log/messages
EOF

sudo systemctl restart rsyslog

Monitoring and Alerting

Log Rotation Monitoring Script

# Create monitoring script for log rotation
sudo tee /root/scripts/monitor_log_rotation.sh << 'EOF'
#!/bin/bash
# Monitor Log Rotation Status

REPORT_FILE="/var/log/admin/log_rotation_status_$(date +%Y%m%d).log"
ALERT_EMAIL="[email protected]"
WARN_SIZE_MB=5000  # Warn if log directory exceeds 5GB

mkdir -p /var/log/admin

exec > >(tee -a "$REPORT_FILE")

echo "========================================"
echo "LOG ROTATION STATUS REPORT"
echo "Date: $(date)"
echo "========================================"
echo ""

# Check logrotate status file
echo "=== Last Rotation Times ==="
tail -20 /var/lib/logrotate/status
echo ""

# Check for large unrotated logs
echo "=== Large Log Files (>100MB) ==="
find /var/log -type f -size +100M -exec ls -lh {} \; | \
    awk '{print $9, $5}' | sort -k2 -hr
echo ""

# Check disk usage by log directory
echo "=== Disk Usage by Log Directory ==="
du -sh /var/log/*/ 2>/dev/null | sort -hr | head -20
echo ""

# Check total log directory size
TOTAL_SIZE_KB=$(du -s /var/log | awk '{print $1}')
TOTAL_SIZE_MB=$((TOTAL_SIZE_KB / 1024))

echo "Total /var/log size: ${TOTAL_SIZE_MB}MB"

if [ "$TOTAL_SIZE_MB" -gt "$WARN_SIZE_MB" ]; then
    echo "WARNING: Log directory size exceeds threshold!"
    echo "Log directory size: ${TOTAL_SIZE_MB}MB (threshold: ${WARN_SIZE_MB}MB)" | \
        mail -s "Log Size Alert" "$ALERT_EMAIL"
fi
echo ""

# Check for rotation failures
echo "=== Recent Logrotate Errors ==="
grep -i "error" /var/log/syslog | grep logrotate | tail -10
echo ""

# Check for logs that haven't been rotated recently
echo "=== Logs Not Rotated in 7+ Days ==="
find /var/log -name "*.log" -type f -mtime +7 -exec ls -lh {} \; | \
    awk '{print $9, $5, $6, $7, $8}'
echo ""

# Verify logrotate is scheduled
echo "=== Logrotate Schedule Status ==="
if systemctl is-enabled logrotate.timer &>/dev/null; then
    echo "Logrotate timer: ENABLED"
    systemctl status logrotate.timer | grep -E "Active|Trigger"
elif [ -f /etc/cron.daily/logrotate ]; then
    echo "Logrotate cron job: EXISTS"
    ls -l /etc/cron.daily/logrotate
else
    echo "WARNING: Logrotate scheduling not found!"
    echo "Logrotate not scheduled!" | mail -s "Logrotate Alert" "$ALERT_EMAIL"
fi
echo ""

echo "========================================"
echo "REPORT COMPLETED: $(date)"
echo "========================================"

EOF

sudo chmod +x /root/scripts/monitor_log_rotation.sh

# Schedule weekly monitoring
echo "0 8 * * 1 /root/scripts/monitor_log_rotation.sh | mail -s 'Weekly Log Rotation Report' [email protected]" | sudo crontab -

Disk Space Monitoring

# Create disk space monitoring for logs
sudo tee /root/scripts/monitor_log_disk_usage.sh << 'EOF'
#!/bin/bash
# Monitor Log Disk Usage

ALERT_EMAIL="[email protected]"
WARN_PERCENT=80
CRIT_PERCENT=90

# Check log partition usage
LOG_USAGE=$(df -h /var/log | tail -1 | awk '{print $5}' | sed 's/%//')

if [ "$LOG_USAGE" -ge "$CRIT_PERCENT" ]; then
    echo "CRITICAL: Log partition usage at ${LOG_USAGE}%" | \
        mail -s "CRITICAL: Log Disk Usage Alert" "$ALERT_EMAIL"
elif [ "$LOG_USAGE" -ge "$WARN_PERCENT" ]; then
    echo "WARNING: Log partition usage at ${LOG_USAGE}%" | \
        mail -s "WARNING: Log Disk Usage Alert" "$ALERT_EMAIL"
fi

# Log largest files for investigation
if [ "$LOG_USAGE" -ge "$WARN_PERCENT" ]; then
    echo "Top 20 largest log files:" > /tmp/large_logs.txt
    find /var/log -type f -exec du -h {} \; | sort -hr | head -20 >> /tmp/large_logs.txt
    mail -s "Large Log Files Report" "$ALERT_EMAIL" < /tmp/large_logs.txt
    rm /tmp/large_logs.txt
fi

EOF

sudo chmod +x /root/scripts/monitor_log_disk_usage.sh

# Schedule hourly disk usage checks
echo "0 * * * * /root/scripts/monitor_log_disk_usage.sh" | sudo crontab -

Log Archiving and Compression

Automated Log Archiving

# Create log archiving script
sudo tee /root/scripts/archive_old_logs.sh << 'EOF'
#!/bin/bash
# Archive old logs to long-term storage

ARCHIVE_DIR="/mnt/archive-storage/logs"
SOURCE_DIR="/var/log"
DAYS_BEFORE_ARCHIVE=90
ARCHIVE_NAME="logs-archive-$(date +%Y%m%d).tar.gz"

mkdir -p "$ARCHIVE_DIR"

# Find logs older than threshold
find "$SOURCE_DIR" -name "*.log-*.gz" -mtime +$DAYS_BEFORE_ARCHIVE | \
    tar -czf "$ARCHIVE_DIR/$ARCHIVE_NAME" -T -

# Verify archive created
if [ -f "$ARCHIVE_DIR/$ARCHIVE_NAME" ]; then
    echo "Archive created: $ARCHIVE_DIR/$ARCHIVE_NAME"

    # Calculate checksum
    sha256sum "$ARCHIVE_DIR/$ARCHIVE_NAME" > "$ARCHIVE_DIR/$ARCHIVE_NAME.sha256"

    # Remove archived files from source
    find "$SOURCE_DIR" -name "*.log-*.gz" -mtime +$DAYS_BEFORE_ARCHIVE -delete

    echo "Old logs archived and removed from $SOURCE_DIR"
else
    echo "ERROR: Archive creation failed"
    exit 1
fi

EOF

sudo chmod +x /root/scripts/archive_old_logs.sh

# Schedule monthly archiving
echo "0 2 1 * * /root/scripts/archive_old_logs.sh" | sudo crontab -

Compress Uncompressed Logs

# Compress old uncompressed logs to save space
sudo tee /root/scripts/compress_old_logs.sh << 'EOF'
#!/bin/bash
# Compress uncompressed log files older than 7 days

find /var/log -name "*.log-*" ! -name "*.gz" -type f -mtime +7 | while read logfile; do
    gzip -9 "$logfile"
    echo "Compressed: $logfile"
done

EOF

sudo chmod +x /root/scripts/compress_old_logs.sh

# Schedule daily compression
echo "0 3 * * * /root/scripts/compress_old_logs.sh" | sudo crontab -

Troubleshooting Log Rotation

Common Issues and Solutions

# Debug logrotate configuration
sudo logrotate -d /etc/logrotate.conf

# Force rotation (useful for testing)
sudo logrotate -f /etc/logrotate.conf

# Check logrotate status
sudo cat /var/lib/logrotate/status

# Manually test specific configuration
sudo logrotate -d /etc/logrotate.d/apache2
sudo logrotate -v -f /etc/logrotate.d/apache2

# Check for errors in syslog
sudo grep logrotate /var/log/syslog | tail -20

# Verify file permissions
ls -la /var/log/*.log
ls -la /var/lib/logrotate/status

# Test postrotate scripts manually
/etc/init.d/apache2 reload
systemctl reload nginx

Fix Common Permission Issues

# Fix logrotate state file permissions
sudo chown root:root /var/lib/logrotate/status
sudo chmod 644 /var/lib/logrotate/status

# Fix log directory permissions
sudo chown -R syslog:adm /var/log
sudo chmod 755 /var/log

# Fix individual log file permissions
sudo find /var/log -name "*.log" -exec chmod 640 {} \;

Best Practices Summary

Log Retention Best Practices

1. Define Clear Retention Policies:

   • Security logs: 1-2 years minimum
   • Compliance logs: Per regulatory requirements
   • Application logs: 30-90 days
   • Debug logs: 7-14 days

2. Implement Multi-Tier Storage:

   • Recent logs (0-30 days): Fast SSD storage
   • Archive logs (31-365 days): Slower HDD storage
   • Long-term archive (1+ years): Tape or cold storage

3. Compress Rotated Logs:

   • Use gzip or bzip2 compression
   • Delay compression by one cycle for recent access
   • Compress logs older than 7 days

4. Monitor Disk Usage:

   • Set up alerts at 80% and 90% capacity
   • Regularly review largest log files
   • Adjust retention policies based on usage

5. Test Rotation Regularly:

   • Verify rotation is working
   • Test log restoration from archives
   • Validate postrotate scripts

6. Document Retention Policies:

   • Document why each retention period was chosen
   • Include regulatory requirements
   • Review and update annually

Configuration Checklist

# Create comprehensive configuration checklist
cat > /root/docs/log_rotation_checklist.md << 'EOF'
# Log Rotation Configuration Checklist

## Configuration Files
- [ ] /etc/logrotate.conf reviewed and configured
- [ ] Service-specific configs in /etc/logrotate.d/ created
- [ ] Retention periods match compliance requirements
- [ ] Compression enabled for rotated logs
- [ ] Postrotate scripts tested

## Monitoring
- [ ] Disk usage monitoring configured
- [ ] Rotation status monitoring scheduled
- [ ] Alerts configured for failures
- [ ] Log review processes established

## Testing
- [ ] Logrotate configuration tested with -d flag
- [ ] Forced rotation tested with -f flag
- [ ] Postrotate scripts execute successfully
- [ ] Log files created with correct permissions
- [ ] Services reload/restart correctly after rotation

## Documentation
- [ ] Retention policies documented
- [ ] Compliance requirements identified
- [ ] Rotation schedules documented
- [ ] Recovery procedures documented

## Security
- [ ] Log file permissions set correctly (640 or 600)
- [ ] Log directories protected (755 with appropriate ownership)
- [ ] Sensitive logs encrypted if required
- [ ] Access to logs restricted to authorized personnel

## Archiving
- [ ] Archive storage configured
- [ ] Archive verification process established
- [ ] Archive retention policy defined
- [ ] Archive restoration tested

EOF

Conclusion

Effective log retention and rotation are essential for maintaining system health, enabling security investigations, and meeting compliance requirements. This guide has provided comprehensive coverage of configuring logrotate for various services, implementing compliance-driven retention policies, and establishing monitoring and archiving procedures.

Key Takeaways

1. Balance Retention with Resources: Longer retention provides better forensic capability but consumes more storage. Find the right balance for your requirements.

2. Automate Everything: Manual log management doesn't scale. Automated rotation, compression, and archiving ensure consistency.

3. Compliance Drives Retention: Understand regulatory requirements (PCI-DSS, GDPR, HIPAA) and configure retention accordingly.

4. Monitor Proactively: Don't wait for disk space issues. Monitor rotation status and disk usage continuously.

5. Test Regularly: Verify that rotation works, archives are accessible, and restoration procedures function correctly.

By implementing the configurations and procedures outlined in this guide, you'll establish robust log management practices that support operational needs, security investigations, and compliance requirements while keeping storage usage under control.