ModSecurity (WAF) Configuration on Apache/Nginx

Introduction

Web Application Firewalls (WAFs) provide essential protection for web applications against common attacks including SQL injection, cross-site scripting (XSS), and many other OWASP Top 10 vulnerabilities. ModSecurity, an open-source WAF, has become the de facto standard for protecting web applications by monitoring, logging, and blocking malicious HTTP traffic before it reaches your application.

Originally designed for Apache, ModSecurity now supports Nginx through a dedicated connector, providing flexible deployment options across the most popular web servers. Combined with the OWASP Core Rule Set (CRS), ModSecurity offers comprehensive protection against a wide range of attacks while remaining highly customizable to meet specific application requirements.

This comprehensive guide covers ModSecurity installation, configuration, and optimization for both Apache and Nginx web servers. Whether you're protecting a simple website or a complex web application infrastructure, this guide provides the knowledge and practical techniques needed to implement effective web application security.

Understanding ModSecurity and Security Context

What Is ModSecurity?

ModSecurity is an open-source web application firewall (WAF) that operates as a module within web servers. It inspects HTTP(S) traffic in real-time, applying security rules to detect and prevent attacks before they reach your application.

How ModSecurity Works

ModSecurity operates through several key mechanisms:

1. Request Processing Phases

ModSecurity can inspect and intervene at five phases:

• Phase 1 (Request Headers): Inspects HTTP request headers
• Phase 2 (Request Body): Examines POST data, file uploads
• Phase 3 (Response Headers): Inspects response headers from application
• Phase 4 (Response Body): Examines HTML, JSON responses
• Phase 5 (Logging): Performs logging actions

2. Rule Engine

Rules define patterns to match and actions to take:

SecRule REQUEST_URI "@contains /admin" "id:1,phase:1,deny,status:403,msg:'Admin access denied'"

3. Detection Methods

• Pattern matching: Regular expressions and string matching
• Anomaly scoring: Assigns scores to suspicious patterns
• IP reputation: Blocks known malicious IPs
• Rate limiting: Prevents brute-force and DoS attacks
• Protocol validation: Enforces HTTP RFC compliance

4. Response Actions

• Block: Return error code (403, 404, etc.)
• Redirect: Send to different URL
• Log: Record for analysis
• Pass: Allow but log
• Drop: Silently drop connection

OWASP Core Rule Set (CRS)

The OWASP CRS is a collection of generic attack detection rules:

• Protects against OWASP Top 10 vulnerabilities
• Regular updates for new threats
• Minimal false positives with proper tuning
• Paranoia levels for security vs. compatibility balance
• Community-maintained and supported

Why ModSecurity Matters

ModSecurity provides critical web application protection:

• Prevents common attacks: SQL injection, XSS, CSRF, RCE
• Virtual patching: Protect vulnerable applications before patches available
• Compliance: Helps meet PCI DSS, HIPAA requirements
• Attack visibility: Detailed logging of attack attempts
• Defense in depth: Additional layer beyond application security
• Zero-day protection: Generic rules catch unknown exploits
• Reduces risk: Protects legacy applications

ModSecurity Deployment Modes

Detection Only (Monitoring)

• Rules detect but don't block
• Useful for tuning and testing
• Provides visibility without risk

Blocking (Protection)

• Rules actively block attacks
• Provides real security protection
• Requires proper tuning to avoid false positives

Reverse Proxy

• ModSecurity on separate server
• Protects multiple backend servers
• Centralized security management

Prerequisites

Before configuring ModSecurity, ensure you have:

System Requirements

• Operating System: Linux (Ubuntu, Debian, CentOS, RHEL)
• Web Server: Apache 2.4+ or Nginx 1.10+
• Memory: Minimum 512MB RAM (more for high traffic)
• CPU: Multi-core recommended for high traffic sites
• Disk Space: Adequate space for logs

Required Knowledge

• Web server administration (Apache or Nginx)
• Basic regular expressions
• HTTP protocol fundamentals
• Log analysis skills
• Application-specific knowledge for tuning

Software Requirements

For Apache on Ubuntu/Debian:

sudo apt-get update
sudo apt-get install libapache2-mod-security2

For Apache on CentOS/RHEL:

sudo yum install mod_security

For Nginx:

Nginx requires compilation with ModSecurity module (covered in installation steps).

Important Pre-Configuration Steps

1. Backup configuration: Save current web server config
2. Test environment: Have staging environment for testing
3. Baseline traffic: Understand normal application traffic patterns
4. Monitoring: Ensure log monitoring is in place
5. Plan for tuning: Allocate time for false positive remediation

Step-by-Step ModSecurity Configuration

Step 1: Install ModSecurity for Apache

Ubuntu/Debian:

sudo apt-get update
sudo apt-get install libapache2-mod-security2

Enable the module:

sudo a2enmod security2
sudo systemctl restart apache2

Verify installation:

apachectl -M | grep security

Should show: security2_module (shared)

CentOS/RHEL:

sudo yum install mod_security
sudo systemctl restart httpd

Step 2: Install ModSecurity for Nginx

Option 1: Install from Repository (if available)

sudo apt-get install libnginx-mod-security

Option 2: Compile Nginx with ModSecurity

This is more complex but provides latest versions:

Install dependencies:

sudo apt-get install git build-essential libpcre3 libpcre3-dev zlib1g zlib1g-dev \
    libssl-dev libgd-dev libxml2 libxml2-dev uuid-dev

Download and compile ModSecurity library:

cd /opt
sudo git clone --depth 1 -b v3/master --single-branch https://github.com/SpiderLabs/ModSecurity
cd ModSecurity
sudo git submodule init
sudo git submodule update
sudo ./build.sh
sudo ./configure
sudo make
sudo make install

Download ModSecurity-nginx connector:

cd /opt
sudo git clone --depth 1 https://github.com/SpiderLabs/ModSecurity-nginx.git

Compile Nginx with ModSecurity:

cd /opt
sudo wget http://nginx.org/download/nginx-1.24.0.tar.gz
sudo tar -xzvf nginx-1.24.0.tar.gz
cd nginx-1.24.0
sudo ./configure --add-dynamic-module=../ModSecurity-nginx \
    --with-http_ssl_module \
    --with-http_v2_module
sudo make modules
sudo cp objs/ngx_http_modsecurity_module.so /usr/share/nginx/modules/

Load module in Nginx:

Add to /etc/nginx/nginx.conf (at the top):

load_module modules/ngx_http_modsecurity_module.so;

Step 3: Configure ModSecurity Basic Settings

For Apache:

Copy the recommended configuration:

sudo cp /etc/modsecurity/modsecurity.conf-recommended /etc/modsecurity/modsecurity.conf

For Nginx:

sudo mkdir -p /etc/nginx/modsec
sudo cp /opt/ModSecurity/modsecurity.conf-recommended /etc/nginx/modsec/modsecurity.conf

Edit the main configuration:

sudo nano /etc/modsecurity/modsecurity.conf  # Apache
# or
sudo nano /etc/nginx/modsec/modsecurity.conf  # Nginx

Key settings to configure:

# Enable ModSecurity (change from DetectionOnly to On)
SecRuleEngine On

# Request body access (needed to inspect POST data)
SecRequestBodyAccess On

# Maximum request body size (50MB)
SecRequestBodyLimit 52428800

# Response body access (inspect responses)
SecResponseBodyAccess On

# Response body MIME types to inspect
SecResponseBodyMimeType text/plain text/html text/xml application/json

# Directory for temporary files
SecTmpDir /tmp/

# Directory for persistent data
SecDataDir /var/cache/modsecurity

# Audit logging
SecAuditEngine RelevantOnly
SecAuditLogRelevantStatus "^(?:5|4(?!04))"
SecAuditLogParts ABIJDEFHZ
SecAuditLogType Serial
SecAuditLog /var/log/modsec_audit.log

# Debug logging (disable in production)
SecDebugLog /var/log/modsec_debug.log
SecDebugLogLevel 0

Step 4: Download and Install OWASP Core Rule Set (CRS)

Download CRS:

cd /opt
sudo git clone https://github.com/coreruleset/coreruleset.git
cd coreruleset
sudo git checkout v4.0/main

Install CRS:

For Apache:

sudo cp -r /opt/coreruleset/rules /etc/modsecurity/
sudo cp /opt/coreruleset/crs-setup.conf.example /etc/modsecurity/crs-setup.conf

For Nginx:

sudo cp -r /opt/coreruleset/rules /etc/nginx/modsec/
sudo cp /opt/coreruleset/crs-setup.conf.example /etc/nginx/modsec/crs-setup.conf

Step 5: Configure ModSecurity in Web Server

For Apache:

Create ModSecurity configuration file:

sudo nano /etc/apache2/mods-enabled/security2.conf

Add:

<IfModule security2_module>
    SecDataDir /var/cache/modsecurity
    IncludeOptional /etc/modsecurity/*.conf
    IncludeOptional /etc/modsecurity/rules/*.conf
</IfModule>

Enable in VirtualHost:

sudo nano /etc/apache2/sites-available/000-default.conf

Add inside <VirtualHost>:

<VirtualHost *:80>
    ServerName example.com
    DocumentRoot /var/www/html

    SecRuleEngine On

    # Your other directives
</VirtualHost>

For Nginx:

Create main ModSecurity include file:

sudo nano /etc/nginx/modsec/main.conf

Add:

Include /etc/nginx/modsec/modsecurity.conf
Include /etc/nginx/modsec/crs-setup.conf
Include /etc/nginx/modsec/rules/*.conf

Enable in Nginx server block:

sudo nano /etc/nginx/sites-available/default

Add inside server block:

server {
    listen 80;
    server_name example.com;

    modsecurity on;
    modsecurity_rules_file /etc/nginx/modsec/main.conf;

    location / {
        root /var/www/html;
        index index.html;
    }
}

Step 6: Test Configuration

For Apache:

sudo apachectl configtest
sudo systemctl restart apache2

For Nginx:

sudo nginx -t
sudo systemctl restart nginx

Verify ModSecurity is working:

curl "http://localhost/?param=<script>alert(1)</script>"

Should be blocked with 403 Forbidden.

Check logs:

sudo tail -f /var/log/modsec_audit.log

Step 7: Set Anomaly Scoring Threshold

Edit CRS setup:

sudo nano /etc/modsecurity/crs-setup.conf  # Apache
# or
sudo nano /etc/nginx/modsec/crs-setup.conf  # Nginx

Configure paranoia level and thresholds:

# Paranoia Level (1-4, higher = more strict)
SecAction \
    "id:900000,\
     phase:1,\
     nolog,\
     pass,\
     t:none,\
     setvar:tx.paranoia_level=1"

# Inbound Anomaly Score Threshold
SecAction \
    "id:900110,\
     phase:1,\
     nolog,\
     pass,\
     t:none,\
     setvar:tx.inbound_anomaly_score_threshold=5"

# Outbound Anomaly Score Threshold
SecAction \
    "id:900111,\
     phase:1,\
     nolog,\
     pass,\
     t:none,\
     setvar:tx.outbound_anomaly_score_threshold=4"

Paranoia levels:

• Level 1: Basic protection, minimal false positives
• Level 2: Extended protection, some tuning needed
• Level 3: Strict protection, significant tuning required
• Level 4: Maximum protection, extensive tuning essential

Step 8: Create Custom Rules

Create custom rules file:

sudo nano /etc/modsecurity/rules/custom.conf  # Apache
# or
sudo nano /etc/nginx/modsec/rules/custom.conf  # Nginx

Example custom rules:

# Block specific user agents
SecRule REQUEST_HEADERS:User-Agent "@contains badbot" \
    "id:1001,phase:1,deny,status:403,msg:'Bad bot blocked'"

# Rate limiting by IP
SecAction "id:1002,phase:1,nolog,pass,initcol:ip=%{REMOTE_ADDR},\
    setvar:ip.requests=+1,expirevar:ip.requests=60"

SecRule IP:REQUESTS "@gt 100" \
    "id:1003,phase:1,deny,status:429,msg:'Rate limit exceeded'"

# Protect admin area
SecRule REQUEST_URI "@beginsWith /admin" \
    "id:1004,phase:1,chain"
SecRule REMOTE_ADDR "!@ipMatch 192.168.1.0/24" \
    "deny,status:403,msg:'Admin access from unauthorized IP'"

# Block SQL injection attempts in custom parameter
SecRule ARGS:search "@detectSQLi" \
    "id:1005,phase:2,deny,status:403,log,msg:'SQL Injection Attempt in search'"

# Whitelist known IP
SecRule REMOTE_ADDR "@ipMatch 203.0.113.10" \
    "id:1006,phase:1,pass,nolog,ctl:ruleEngine=Off"

Advanced ModSecurity Hardening Tips

1. Implement IP Reputation Lists

Use Project Honey Pot HTTP:BL:

SecRule REMOTE_ADDR "@ipMatchFromFile /etc/modsecurity/blacklist-ips.txt" \
    "id:2001,phase:1,deny,status:403,msg:'IP Blacklisted'"

Update list regularly via cron:

0 2 * * * wget -O /etc/modsecurity/blacklist-ips.txt https://example.com/blacklist.txt

2. Implement Geolocation Blocking

Require GeoIP module and database:

sudo apt-get install libapache2-mod-geoip geoip-database

Block by country:

SecGeoLookupDB /usr/share/GeoIP/GeoIP.dat
SecRule REMOTE_ADDR "@geoLookup" "chain,id:2002,drop,msg:'Blocked by country'"
SecRule GEO:COUNTRY_CODE "@streq CN"

3. Implement Advanced Rate Limiting

Per-URI rate limiting:

SecAction "id:2003,phase:1,nolog,pass,\
    initcol:ip=%{REMOTE_ADDR}_%{REQUEST_URI},\
    setvar:ip.uri_requests=+1,\
    expirevar:ip.uri_requests=10"

SecRule IP:URI_REQUESTS "@gt 20" \
    "id:2004,phase:1,deny,status:429,\
    msg:'Too many requests to specific URI'"

4. Implement Session Management

Track authenticated sessions:

SecRule REQUEST_COOKIES:sessionid "@rx ^[a-zA-Z0-9]{32}$" \
    "id:2005,phase:1,pass,nolog,\
    setvar:tx.session_valid=1"

SecRule TX:SESSION_VALID "!@eq 1" \
    "id:2006,phase:1,deny,status:403,\
    msg:'Invalid session ID format'"

5. Enable Response Header Filtering

Remove server information disclosure:

SecRule RESPONSE_HEADERS:Server "@rx (.*)" \
    "id:2007,phase:3,pass,nolog,\
    setvar:tx.server_header=%{MATCHED_VAR}"

Header always set Server "WebServer"

6. Implement Content Security Policy

Add CSP headers via ModSecurity:

Header always set Content-Security-Policy "default-src 'self'; script-src 'self' 'unsafe-inline'"
Header always set X-Content-Type-Options "nosniff"
Header always set X-Frame-Options "SAMEORIGIN"
Header always set X-XSS-Protection "1; mode=block"

7. Create Application-Specific Rules

For WordPress:

# Block XML-RPC attacks
SecRule REQUEST_URI "@streq /xmlrpc.php" \
    "id:3001,phase:1,deny,status:403,msg:'XML-RPC blocked'"

# Protect wp-login
SecRule REQUEST_URI "@beginsWith /wp-login.php" \
    "id:3002,phase:1,chain"
SecRule &ARGS "@gt 3" \
    "deny,status:403,msg:'Excessive login attempts'"

8. Implement Detailed Audit Logging

Configure comprehensive audit logging:

SecAuditEngine RelevantOnly
SecAuditLogRelevantStatus "^(?:5|4(?!04))"
SecAuditLogParts ABCDEFGHIJKZ
SecAuditLogType Concurrent
SecAuditLogStorageDir /var/log/modsec_audit/
SecAuditLogDirMode 0750
SecAuditLogFileMode 0640

Verification and Testing

Verify ModSecurity is Active

For Apache:

curl -I http://localhost

Look for Server header modifications or check error log:

sudo tail /var/log/apache2/error.log | grep ModSecurity

For Nginx:

sudo nginx -V 2>&1 | grep modsecurity

Test Basic Attack Detection

Test XSS detection:

curl "http://localhost/?test=<script>alert(1)</script>"

Should return 403 Forbidden.

Test SQL injection detection:

curl "http://localhost/?id=1' OR '1'='1"

Should be blocked.

Test path traversal:

curl "http://localhost/../../../../etc/passwd"

Should be blocked.

Monitor Audit Logs

Watch audit log in real-time:

sudo tail -f /var/log/modsec_audit.log

Search for specific attacks:

sudo grep "SQL Injection" /var/log/modsec_audit.log

Test False Positive Scenarios

Legitimate requests that might trigger false positives:

1. Complex search queries
2. URL parameters with special characters
3. File uploads
4. API POST requests with JSON

Document and whitelist as needed.

Performance Testing

Benchmark without ModSecurity:

ab -n 1000 -c 10 http://localhost/

Enable ModSecurity and benchmark again:

ab -n 1000 -c 10 http://localhost/

Compare results for performance impact.

Troubleshooting Common Issues

Issue 1: Too Many False Positives

Symptoms: Legitimate requests blocked

Solutions:

1. Identify the rule:

   sudo grep "id" /var/log/modsec_audit.log | tail -20
   

2. Lower paranoia level: Reduce from level 2 to level 1 in crs-setup.conf

3. Whitelist specific rule for URI:

   SecRuleRemoveById 942100
   

4. Create exception:

   SecRule REQUEST_URI "@beginsWith /api/" \
       "id:4001,phase:1,pass,nolog,ctl:ruleRemoveById=942100"
   

Issue 2: Performance Degradation

Symptoms: Slow response times after enabling ModSecurity

Solutions:

1. Disable response body inspection if not needed:

   SecResponseBodyAccess Off
   

2. Reduce request body limit:

   SecRequestBodyLimit 10485760
   

3. Disable audit logging for normal requests:

   SecAuditEngine RelevantOnly
   

4. Use concurrent audit logging:

   SecAuditLogType Concurrent
   

Issue 3: Configuration Errors

Symptoms: Web server won't start

Solutions:

1. Check syntax:

   sudo apachectl configtest  # Apache
   sudo nginx -t              # Nginx
   

2. Review error logs:

   sudo tail -50 /var/log/apache2/error.log
   sudo tail -50 /var/log/nginx/error.log
   

3. Common issues:

   • Missing Include statements
   • Incorrect file paths
   • Duplicate rule IDs
   • Syntax errors in custom rules

Issue 4: Rules Not Loading

Symptoms: Attacks not being blocked

Solutions:

1. Verify SecRuleEngine is On:

   sudo grep "SecRuleEngine" /etc/modsecurity/modsecurity.conf
   

2. Check Include statements: Ensure CRS rules are included

3. Verify file permissions:

   sudo ls -la /etc/modsecurity/rules/
   

4. Test specific rule manually:

   SecRule ARGS "@contains test" \
       "id:9999,phase:2,deny,status:403,msg:'Test rule'"
   

Issue 5: Audit Log Filling Disk

Symptoms: Disk space exhausted

Solutions:

1. Use log rotation:

   sudo nano /etc/logrotate.d/modsecurity
   

   Add:

   /var/log/modsec_audit.log {
       daily
       rotate 7
       compress
       delaycompress
       notifempty
       create 640 www-data www-data
       postrotate
           systemctl reload apache2 > /dev/null
       endscript
   }
   

2. Reduce audit logging:

   SecAuditEngine RelevantOnly
   SecAuditLogRelevantStatus "^5"
   

Issue 6: JSON API Issues

Symptoms: JSON APIs being blocked incorrectly

Solutions:

1. Ensure JSON inspection is configured:

   SecRule REQUEST_HEADERS:Content-Type "application/json" \
       "id:4002,phase:1,pass,nolog,ctl:requestBodyProcessor=JSON"
   

2. Whitelist API endpoints if necessary:

   SecRule REQUEST_URI "@beginsWith /api/" \
       "id:4003,phase:1,pass,nolog,ctl:ruleRemoveById=920000-920999"
   

Best Practices for ModSecurity Management

1. Deployment Strategy

• Start in DetectionOnly mode: Monitor without blocking initially
• Tune for your application: Spend time eliminating false positives
• Gradual rollout: Enable blocking on non-critical services first
• Have rollback plan: Be prepared to disable if issues arise
• Monitor continuously: Watch logs after enabling blocking

2. Rule Management

• Use CRS as baseline: Don't reinvent the wheel
• Keep CRS updated: Regular updates for new threats
• Document customizations: Maintain records of all custom rules
• Version control: Track changes to ModSecurity configuration
• Test before production: Always test rule changes in staging

3. Tuning Process

1. Enable DetectionOnly mode
2. Generate normal application traffic
3. Review logs for false positives
4. Create exceptions for legitimate traffic
5. Enable blocking mode
6. Monitor for new false positives
7. Iterate and refine

4. Performance Optimization

• Disable unnecessary features: Turn off what you don't need
• Use concurrent audit logging: Better performance for high traffic
• Limit body inspection: Set reasonable size limits
• Optimize custom rules: Efficient regular expressions
• Consider hardware: More CPU/RAM for high-traffic sites

5. Security Hardening

• Use appropriate paranoia level: Balance security vs. usability
• Implement rate limiting: Prevent brute-force attacks
• Block known bad IPs: Use reputation lists
• Monitor for anomalies: Set up alerting for attack patterns
• Regular security audits: Review configuration quarterly

6. Logging and Monitoring

• Centralize logs: Send to SIEM or log aggregation platform
• Set up alerts: Notify on high-severity events
• Regular log review: Analyze attack patterns
• Compliance logging: Retain logs per regulatory requirements
• Performance metrics: Track ModSecurity overhead

7. Incident Response

• Document procedures: How to respond to attacks
• Emergency disablement: Know how to quickly disable if needed
• Attack analysis: Process for investigating blocked requests
• Rule updates: Procedure for implementing emergency rules
• Communication plan: Who to notify during incidents

Conclusion

ModSecurity provides robust, enterprise-grade web application firewall protection that defends against a wide range of attacks including OWASP Top 10 vulnerabilities. Combined with the OWASP Core Rule Set, ModSecurity offers comprehensive security that adapts to evolving threats through regular updates and community contributions.

Key takeaways:

• Essential protection: WAF layer critical for web application security
• Flexible deployment: Works with Apache and Nginx
• OWASP CRS: Provides mature, tested ruleset for common attacks
• Customizable: Adapt to specific application requirements
• Tuning required: Investment in tuning pays off in accurate protection
• Continuous monitoring: Ongoing management ensures effectiveness

ModSecurity is particularly valuable for:

• Protecting against OWASP Top 10 vulnerabilities
• Virtual patching of vulnerable applications
• Compliance with PCI DSS and other frameworks
• Gaining visibility into attack patterns
• Defense-in-depth security strategy
• Protecting legacy applications

Remember that ModSecurity is most effective when:

• Properly tuned to your application
• Regularly updated with latest rules
• Continuously monitored for alerts
• Part of comprehensive security strategy
• Backed by incident response procedures

Start with detection-only mode, invest time in proper tuning, and gradually enable blocking as confidence grows. With proper implementation and maintenance, ModSecurity becomes an invaluable security control that protects your web applications from a vast array of threats while providing detailed visibility into attack attempts. Combined with secure coding practices, regular vulnerability assessments, and comprehensive monitoring, ModSecurity forms a critical component of modern web application security.