Detecting Server Under Attack

Introduction

Cyber attacks on servers are increasing in frequency and sophistication. From distributed denial-of-service (DDoS) attacks to brute force attempts and malware infections, servers face constant threats from malicious actors. Early detection is crucial to minimize damage, prevent data breaches, and maintain service availability.

This comprehensive guide provides system administrators and security professionals with practical, command-line tools and techniques to detect when a server is under attack. You'll learn to identify various attack patterns, analyze suspicious activity, and implement real-time detection mechanisms using built-in Linux tools.

Understanding attack indicators and having diagnostic procedures in place is essential for anyone managing production servers. This guide covers everything from recognizing common attack patterns to implementing automated detection systems that alert you to threats in real-time.

Understanding Server Attacks

Common Attack Types

Servers commonly face these types of attacks:

  1. DDoS (Distributed Denial of Service): Overwhelming server resources with traffic
  2. Brute Force: Automated password guessing attempts
  3. Port Scanning: Reconnaissance to identify vulnerabilities
  4. Malware/Rootkit: Malicious software installation
  5. SQL Injection: Database exploitation through web applications
  6. Zero-Day Exploits: Attacks on unknown vulnerabilities
  7. Resource Exhaustion: Consuming CPU, memory, or disk
  8. Application Layer Attacks: Targeting specific services

Attack Indicators

Common signs your server may be under attack:

  • Unusual network traffic spikes
  • High CPU or memory usage without explanation
  • Excessive failed login attempts
  • Unknown processes running
  • Unexpected network connections
  • Disk space rapidly filling
  • Service degradation or crashes
  • Firewall logs showing blocked attempts
  • Unauthorized file modifications

Initial Attack Detection

Quick Security Assessment

Run these commands immediately when you suspect an attack:

# Check current logged-in users
w
who
last | head -20

# Check active network connections
ss -tunap | wc -l
netstat -an | grep ESTABLISHED | wc -l

# Check for unusual processes
ps aux --sort=-%cpu | head -15
ps aux --sort=-%mem | head -15

# Check recent authentication attempts
tail -100 /var/log/auth.log | grep -i "failed\|refused"
journalctl -u sshd --since "10 minutes ago" | grep -i fail

# Check system load
uptime
top -bn1 | head -15

# Check disk activity
iostat -x 1 3

Step 1: Network Traffic Analysis

Monitoring Active Connections

Analyze current network connections for suspicious patterns:

# Count connections by state
ss -s
netstat -an | awk '{print $6}' | sort | uniq -c | sort -rn

# List all active connections
ss -tunap
netstat -tunap

# Count connections by source IP
netstat -ntu | awk '{print $5}' | cut -d: -f1 | sort | uniq -c | sort -rn | head -20

# Identify IPs with most connections
ss -tn | awk '{print $4}' | cut -d: -f1 | sort | uniq -c | sort -rn | head -20

# Check for SYN flood attack
netstat -an | grep SYN_RECV | wc -l
ss -ant | grep SYN-RECV | wc -l

# Monitor connection rate in real-time
watch -n 1 'ss -s'

Suspicious Indicators:

  • Single IP with hundreds/thousands of connections
  • Excessive SYN_RECV states (>100)
  • Many connections to unusual ports
  • Connections from unexpected countries/regions
  • Rapid connection establishment/termination

Network Traffic Volume Analysis

Monitor bandwidth usage:

# Real-time bandwidth monitoring
iftop -i eth0
iftop -P -i eth0  # Show ports

# Interface statistics
ip -s link
ifconfig

# Network usage by process
nethogs eth0

# Packet statistics
watch -n 1 'cat /proc/net/dev'

# Detailed interface statistics
sar -n DEV 1 10

Analyzing Traffic with tcpdump

Capture and analyze network packets:

# Capture traffic on interface
tcpdump -i eth0 -n

# Capture specific port traffic
tcpdump -i eth0 port 80 -n

# Capture traffic from specific IP
tcpdump -i eth0 src 192.168.1.100 -n

# Count packets by source IP
tcpdump -i eth0 -n | awk '{print $3}' | cut -d. -f1-4 | sort | uniq -c | sort -rn

# Detect SYN flood
tcpdump -i eth0 'tcp[tcpflags] & (tcp-syn) != 0' -c 1000 -n

# Save capture for analysis
tcpdump -i eth0 -w /tmp/capture.pcap -c 10000

# Analyze saved capture
tcpdump -r /tmp/capture.pcap | awk '{print $3}' | cut -d. -f1-4 | sort | uniq -c | sort -rn

Detecting Port Scans

Identify reconnaissance attempts:

# Monitor new connections
watch -n 1 'ss -tan | grep ESTABLISHED | wc -l'

# Check for connections to multiple ports from same IP
tail -1000 /var/log/syslog | grep "UFW BLOCK" | awk '{print $11}' | sort | uniq -c | sort -rn

# Iptables log analysis
grep "IN=.*OUT=.*" /var/log/kern.log | tail -100

# Detect rapid connection attempts
awk '/DPT/ {print $12}' /var/log/messages | cut -d= -f2 | sort | uniq -c | sort -rn

Step 2: Authentication Attack Detection

Monitoring Failed Login Attempts

Brute force attacks leave clear signatures:

# Recent failed SSH attempts
grep "Failed password" /var/log/auth.log | tail -50

# Count failed attempts by IP
grep "Failed password" /var/log/auth.log | awk '{print $(NF-3)}' | sort | uniq -c | sort -rn

# Failed attempts in last hour
grep "Failed password" /var/log/auth.log | grep "$(date '+%b %e %H')"

# Using journalctl for systemd
journalctl -u sshd | grep "Failed password" | tail -50

# Count failed attempts by username
grep "Failed password" /var/log/auth.log | awk '{print $9}' | sort | uniq -c | sort -rn

# Check for dictionary attacks (invalid users)
grep "Invalid user" /var/log/auth.log | tail -50
grep "Invalid user" /var/log/auth.log | awk '{print $8}' | sort | uniq -c | sort -rn

# Monitor in real-time
tail -f /var/log/auth.log | grep --line-buffered "Failed\|Invalid"

Attack Indicators:

  • Multiple failed attempts from same IP (>10)
  • Failed attempts for multiple usernames
  • Failed attempts for non-existent users
  • Attempts from multiple IPs simultaneously
  • Failed attempts at regular intervals (automated)

Checking Successful Suspicious Logins

# Recent successful logins
grep "Accepted" /var/log/auth.log | tail -50

# Login history
last | head -50
last -f /var/log/wtmp | head -50

# Failed login attempts
lastb | head -50
last -f /var/log/btmp | head -50

# Currently logged-in users
w
who -a

# Check for unusual login times
last | grep -v "$(date '+%a %b %e')"

# Logins from unusual locations
last | awk '{print $3}' | grep -v "^$\|wtmp" | sort | uniq

# Check user login history
last username | head -20

Analyzing Authentication Patterns

# Hourly failed login count
grep "Failed password" /var/log/auth.log | awk '{print $1,$2,$3}' | sort | uniq -c

# Geographic analysis of IPs (requires geoip-bin)
grep "Failed password" /var/log/auth.log | awk '{print $(NF-3)}' | sort -u | xargs -I {} geoiplookup {}

# Check for password spray attacks
grep "Failed password" /var/log/auth.log | awk '{print $9}' | sort | uniq -c | awk '$1 < 5 {print $2}' | wc -l

# Time-based analysis
for hour in {00..23}; do
    echo -n "$hour:00 - "
    grep "Failed password" /var/log/auth.log | grep " $hour:" | wc -l
done

Step 3: Process and System Analysis

Identifying Malicious Processes

Look for suspicious running processes:

# Processes by CPU usage
ps aux --sort=-%cpu | head -20

# Processes by memory usage
ps aux --sort=-%mem | head -20

# Processes running as unexpected users
ps aux | grep -v "^root\|^www-data\|^mysql\|^postgres" | less

# Recently started processes
ps -eo pid,user,lstart,cmd --sort=lstart | tail -20

# Processes with network connections
lsof -i -n -P | grep ESTABLISHED

# Hidden processes (possible rootkit)
ps aux | wc -l
ls /proc | grep "^[0-9]" | wc -l
# Numbers should match

# Check for processes listening on ports
lsof -i -P | grep LISTEN
ss -tlnp

# Processes with unusual names
ps aux | grep -E "\[.*\]" | grep -v "^\[.*\]$"

# Check process executable paths
ls -la /proc/*/exe 2>/dev/null | grep deleted

Suspicious Indicators:

  • Processes with random character names
  • Processes running from /tmp or /dev/shm
  • Processes with no parent (PPID = 1 unexpectedly)
  • Processes consuming excessive resources
  • Unknown processes listening on ports
  • Deleted executables still running

Analyzing Process Behavior

# Detailed process information
ps -fp <PID>
cat /proc/<PID>/cmdline
cat /proc/<PID>/status

# Process network activity
lsof -p <PID> -i

# Process open files
lsof -p <PID>

# Process working directory
ls -la /proc/<PID>/cwd

# Process start time
ps -p <PID> -o lstart=

# Trace process system calls
strace -p <PID> -s 1000 -e trace=network

# Check process binary
file /proc/<PID>/exe
md5sum /proc/<PID>/exe

Checking Scheduled Tasks

Attackers often use cron for persistence:

# User crontabs
for user in $(cut -f1 -d: /etc/passwd); do
    echo "Crontab for $user:"
    crontab -u $user -l 2>/dev/null
done

# System crontabs
cat /etc/crontab
ls -la /etc/cron.*/*
cat /etc/cron.d/*

# Systemd timers
systemctl list-timers --all

# At jobs
atq
for job in $(atq | awk '{print $1}'); do
    echo "Job $job:"
    at -c $job
done

# Check for suspicious entries
grep -r "wget\|curl\|nc\|/tmp" /etc/cron* /var/spool/cron/

Step 4: File System Analysis

Detecting Unauthorized File Changes

Monitor for modified or new files:

# Recently modified files
find / -type f -mtime -1 -ls 2>/dev/null | grep -v "/proc\|/sys\|/dev"

# Recently created files in suspicious locations
find /tmp /var/tmp /dev/shm -type f -mtime -1 -ls 2>/dev/null

# Large files created recently
find / -type f -size +100M -mtime -1 -ls 2>/dev/null

# Files modified in last hour
find / -type f -mmin -60 -ls 2>/dev/null | grep -v "/proc\|/sys"

# Hidden files
find / -name ".*" -type f -ls 2>/dev/null | head -50

# SUID files
find / -perm -4000 -type f -ls 2>/dev/null

# World-writable files
find / -type f -perm -002 -ls 2>/dev/null | grep -v "/proc\|/sys"

# Check critical system files
stat /bin/bash /bin/ls /usr/bin/wget /usr/bin/curl

Analyzing Web Directories

Web directories are common attack targets:

# Recently modified web files
find /var/www -type f -mtime -7 -ls

# PHP/executable files in upload directories
find /var/www -name "*.php" -o -name "*.sh" -o -name "*.py" | grep -i upload

# Suspicious file names
find /var/www -type f -name "*.php.*" -o -name "*shell*" -o -name "*backdoor*"

# Files with unusual permissions
find /var/www -type f -perm -111 -ls

# Check for web shells
grep -r "eval\|base64_decode\|system\|exec\|shell_exec" /var/www --include="*.php" | head -20

Checking System Binaries

Verify critical binaries haven't been replaced:

# Install debsums for Debian/Ubuntu
apt install debsums
debsums -c

# Install rpm -V for CentOS/RHEL
rpm -Va | grep '^..5'

# Check common binary hashes
md5sum /bin/ls /bin/ps /bin/netstat /usr/bin/top
# Compare with known good values

# Check for unusual binaries
ls -lah /bin /usr/bin /sbin /usr/sbin | grep "$(date +%Y-%m-%d)"

Step 5: Log Analysis

System Log Examination

System logs reveal attack patterns:

# Recent critical errors
journalctl -p err -n 100

# Kernel messages
dmesg | tail -100
dmesg | grep -i "segfault\|error\|attack"

# Authentication logs
tail -200 /var/log/auth.log
grep -i "fail\|error\|refused" /var/log/auth.log | tail -100

# System log
tail -200 /var/log/syslog
tail -200 /var/log/messages

# Security log
tail -100 /var/log/secure

# Check for log tampering
ls -lah /var/log/*.log
# Look for gaps in modification times or empty logs

Web Server Log Analysis

Analyze web server logs for attack patterns:

# Top requesting IPs
awk '{print $1}' /var/log/nginx/access.log | sort | uniq -c | sort -rn | head -20

# Requests per minute
awk '{print $4}' /var/log/nginx/access.log | cut -d: -f1-3 | uniq -c | tail -20

# 404 errors (probing)
grep " 404 " /var/log/nginx/access.log | tail -100

# Failed login attempts to web apps
grep "wp-login\|admin\|login" /var/log/nginx/access.log | tail -100

# SQL injection attempts
grep -i "select\|union\|insert\|update" /var/log/nginx/access.log | tail -50

# Directory traversal attempts
grep "\.\." /var/log/nginx/access.log | tail -50

# User agent analysis (detect bots)
awk -F\" '{print $6}' /var/log/nginx/access.log | sort | uniq -c | sort -rn | head -20

# POST request analysis
grep "POST" /var/log/nginx/access.log | tail -100

# Large requests (potential upload attacks)
awk '{print $10}' /var/log/nginx/access.log | sort -rn | head -20

Firewall Log Analysis

# UFW blocked connections
grep "UFW BLOCK" /var/log/syslog | tail -100

# Count blocked IPs
grep "UFW BLOCK" /var/log/syslog | awk '{print $11}' | cut -d= -f2 | sort | uniq -c | sort -rn

# Blocked destination ports
grep "UFW BLOCK" /var/log/syslog | awk '{print $13}' | cut -d= -f2 | sort | uniq -c | sort -rn

# iptables logs
grep "iptables" /var/log/kern.log | tail -100

# Fail2ban actions
tail -100 /var/log/fail2ban.log
grep "Ban\|Unban" /var/log/fail2ban.log | tail -50

Step 6: Malware and Rootkit Detection

Running Security Scanners

Use specialized tools to detect malware:

# Install and run rkhunter
apt install rkhunter
rkhunter --update
rkhunter --check --skip-keypress

# Install and run chkrootkit
apt install chkrootkit
chkrootkit

# ClamAV antivirus scan
apt install clamav clamav-daemon
freshclam
clamscan -r /home /var/www

# Linux Malware Detect
cd /tmp
wget http://www.rfxn.com/downloads/maldetect-current.tar.gz
tar -xzf maldetect-current.tar.gz
cd maldetect-*
./install.sh
maldet --update
maldet -a /var/www

Checking for Rootkits

# Check loaded kernel modules
lsmod
lsmod | grep -v "Module\|^$"

# Compare with known modules
lsmod > /tmp/current_modules.txt
# Compare with baseline

# Check for hidden kernel modules
cat /proc/modules
diff <(lsmod | cut -d' ' -f1 | tail -n +2) <(cat /proc/modules | cut -d' ' -f1)

# Check for LKM rootkits
ls -la /lib/modules/$(uname -r)/kernel/
modinfo suspicious_module_name

# Check system call table
cat /proc/kallsyms | grep sys_call_table

Network-Based Detection

# Check for hidden network connections
ss -tunap vs netstat -tunap

# Check for packet sniffers
lsof -i -n -P | grep -i raw
ip link | grep PROMISC

# Check DNS queries
tcpdump -i any port 53 -n

# Check for C2 (Command & Control) beaconing
tcpdump -i eth0 -n | grep -E "beacon|c2|cnc"

Step 7: Resource Abuse Detection

Cryptocurrency Mining Detection

Cryptominers are increasingly common:

# High CPU processes
ps aux --sort=-%cpu | head -10

# Check for known miner processes
ps aux | grep -i "xmrig\|minerd\|cpuminer\|ethminer"

# Check for mining pools
netstat -anp | grep -E "stratum|pool|mining"

# Check common mining ports
ss -tunap | grep -E ":3333|:4444|:5555|:7777|:8888|:14444"

# Monitor CPU usage pattern
top -b -d 1 -n 60 | grep "Cpu(s)"
# Miners typically use 90-100% CPU constantly

DDoS Participation Detection

Check if your server is being used for attacks:

# Outbound connection analysis
ss -tunap | grep ESTABLISHED | awk '{print $5}' | cut -d: -f1 | sort | uniq -c | sort -rn

# High packet rate outbound
iptables -L OUTPUT -v -n

# Check for SYN flood participation
tcpdump -i eth0 'tcp[tcpflags] & (tcp-syn) != 0' -c 100 -n | awk '{print $5}' | cut -d. -f1-4 | sort | uniq -c

# Bandwidth by destination
iftop -P -o destination

Real-Time Monitoring and Alerting

Creating Attack Detection Script

cat > /usr/local/bin/attack-monitor.sh << 'EOF'
#!/bin/bash

LOG_FILE="/var/log/attack-monitor.log"
ALERT_EMAIL="[email protected]"

# Function to send alert
send_alert() {
    echo "$(date): $1" >> "$LOG_FILE"
    echo "$1" | mail -s "Security Alert: $(hostname)" "$ALERT_EMAIL"
}

# Check failed SSH attempts
FAILED_SSH=$(grep "Failed password" /var/log/auth.log | grep "$(date '+%b %e %H')" | wc -l)
if [ "$FAILED_SSH" -gt 50 ]; then
    send_alert "High number of failed SSH attempts: $FAILED_SSH in last hour"
fi

# Check connection count
CONN_COUNT=$(ss -tan | grep ESTABLISHED | wc -l)
if [ "$CONN_COUNT" -gt 1000 ]; then
    send_alert "Excessive connections detected: $CONN_COUNT"
fi

# Check for port scans
SCAN_ATTEMPTS=$(grep "UFW BLOCK" /var/log/syslog | grep "$(date '+%b %e %H')" | wc -l)
if [ "$SCAN_ATTEMPTS" -gt 100 ]; then
    send_alert "Possible port scan detected: $SCAN_ATTEMPTS blocked attempts"
fi

# Check CPU usage
CPU_USAGE=$(top -bn1 | grep "Cpu(s)" | awk '{print $2}' | cut -d'%' -f1)
if (( $(echo "$CPU_USAGE > 90" | bc -l) )); then
    TOP_PROCESS=$(ps aux --sort=-%cpu | head -2 | tail -1)
    send_alert "High CPU usage: $CPU_USAGE%. Top process: $TOP_PROCESS"
fi

# Check for new SUID files
find / -perm -4000 -type f -mtime -1 2>/dev/null > /tmp/new_suid.txt
if [ -s /tmp/new_suid.txt ]; then
    send_alert "New SUID files detected: $(cat /tmp/new_suid.txt)"
fi
EOF

chmod +x /usr/local/bin/attack-monitor.sh

# Run every 5 minutes
echo "*/5 * * * * /usr/local/bin/attack-monitor.sh" | crontab -

Network Traffic Monitoring

cat > /usr/local/bin/traffic-monitor.sh << 'EOF'
#!/bin/bash

THRESHOLD=1000  # Connections threshold
INTERFACE="eth0"

# Monitor new connections
while true; do
    CURRENT=$(ss -tan | grep ESTABLISHED | wc -l)

    if [ "$CURRENT" -gt "$THRESHOLD" ]; then
        echo "$(date): High connection count: $CURRENT" >> /var/log/traffic-monitor.log

        # Log top IPs
        ss -tn | awk '{print $4}' | cut -d: -f1 | sort | uniq -c | sort -rn | head -10 >> /var/log/traffic-monitor.log
    fi

    sleep 60
done
EOF

chmod +x /usr/local/bin/traffic-monitor.sh

Automated IP Blocking

cat > /usr/local/bin/auto-block.sh << 'EOF'
#!/bin/bash

THRESHOLD=20
LOG="/var/log/auth.log"

# Get IPs with excessive failed attempts
grep "Failed password" "$LOG" | \
    grep "$(date '+%b %e')" | \
    awk '{print $(NF-3)}' | \
    sort | uniq -c | \
    awk -v threshold="$THRESHOLD" '$1 > threshold {print $2}' | \
while read IP; do
    # Check if already blocked
    if ! iptables -L INPUT -n | grep -q "$IP"; then
        echo "$(date): Blocking $IP" >> /var/log/auto-block.log
        iptables -I INPUT -s "$IP" -j DROP

        # Send notification
        echo "Blocked IP: $IP after multiple failed attempts" | \
            mail -s "Auto-blocked IP" [email protected]
    fi
done
EOF

chmod +x /usr/local/bin/auto-block.sh

Response Actions

Immediate Containment

When attack is confirmed:

# Block attacking IP immediately
iptables -I INPUT -s ATTACKING_IP -j DROP

# Block entire subnet if needed
iptables -I INPUT -s 192.168.1.0/24 -j DROP

# Limit connection rate
iptables -A INPUT -p tcp --dport 80 -m limit --limit 25/minute --limit-burst 100 -j ACCEPT

# Kill suspicious processes
kill -9 <PID>

# Disable compromised user account
usermod -L username
passwd -l username

# Disconnect user sessions
pkill -u username

Evidence Collection

# Create incident directory
INCIDENT_DIR="/root/incident-$(date +%Y%m%d-%H%M%S)"
mkdir -p "$INCIDENT_DIR"

# Collect system state
ps auxf > "$INCIDENT_DIR/processes.txt"
ss -tunap > "$INCIDENT_DIR/connections.txt"
iptables -L -n -v > "$INCIDENT_DIR/firewall.txt"
last > "$INCIDENT_DIR/logins.txt"
df -h > "$INCIDENT_DIR/disk.txt"
free -h > "$INCIDENT_DIR/memory.txt"

# Collect logs
cp /var/log/auth.log "$INCIDENT_DIR/"
cp /var/log/syslog "$INCIDENT_DIR/"
journalctl -xe > "$INCIDENT_DIR/journal.txt"

# Collect network state
ip addr > "$INCIDENT_DIR/network.txt"
ip route > "$INCIDENT_DIR/routes.txt"

# Create tarball
tar -czf "incident-$(date +%Y%m%d-%H%M%S).tar.gz" "$INCIDENT_DIR"

Prevention and Hardening

Implementing Fail2Ban

# Install fail2ban
apt install fail2ban

# Configure SSH jail
cat > /etc/fail2ban/jail.local << 'EOF'
[sshd]
enabled = true
port = ssh
filter = sshd
logpath = /var/log/auth.log
maxretry = 3
bantime = 3600
findtime = 600
EOF

# Start and enable
systemctl start fail2ban
systemctl enable fail2ban

# Check status
fail2ban-client status
fail2ban-client status sshd

Rate Limiting

# SSH rate limiting with iptables
iptables -A INPUT -p tcp --dport 22 -m state --state NEW -m recent --set
iptables -A INPUT -p tcp --dport 22 -m state --state NEW -m recent --update --seconds 60 --hitcount 4 -j DROP

# HTTP rate limiting
iptables -A INPUT -p tcp --dport 80 -m limit --limit 25/minute --limit-burst 100 -j ACCEPT

Monitoring Setup

# Install monitoring tools
apt install sysstat aide tripwire

# Configure AIDE (file integrity)
aideinit
cp /var/lib/aide/aide.db.new /var/lib/aide/aide.db

# Daily AIDE check
echo "0 2 * * * /usr/bin/aide --check | mail -s 'AIDE Report' [email protected]" | crontab -

Conclusion

Detecting server attacks requires vigilance, proper tools, and systematic analysis. Key takeaways:

  1. Monitor continuously: Implement automated monitoring and alerting
  2. Know your baseline: Understand normal server behavior
  3. Act quickly: Early detection minimizes damage
  4. Collect evidence: Document attacks for analysis and improvement
  5. Layer defenses: Use multiple detection and prevention methods
  6. Stay updated: Keep systems patched and security tools current
  7. Review logs regularly: Don't wait for alerts to check logs

Regular security audits, proper hardening, and proactive monitoring significantly reduce attack success rates. When attacks do occur, rapid detection and response limit their impact. Keep these diagnostic commands handy and practice using them in non-emergency situations to build familiarity and speed when facing real threats.