OSSEC Host-Based Intrusion Detection

OSSEC is a comprehensive host-based intrusion detection system that provides real-time monitoring of file integrity, system logs, rootkits, and active response capabilities. Unlike network-based IDS systems, OSSEC operates on individual servers and endpoints, detecting attacks from internal compromise, privilege escalation, and system tampering. This guide covers installing OSSEC in server and agent configurations, creating and managing detection rules, configuring syscheck for file integrity monitoring, implementing active response mechanisms, and setting up email alerts for critical security events.

Table of Contents

• System Requirements
• Installation
• OSSEC Architecture
• Server Installation
• Agent Installation
• Rule Creation and Management
• Syscheck Configuration
• Rootcheck Setup
• Active Response
• Email Alerts
• Log Aggregation
• Conclusion

System Requirements

OSSEC requires minimal system resources and can run on older hardware. Ensure these prerequisites:

• Linux kernel 2.6 or newer
• At least 512 MB RAM (1 GB+ for large deployments)
• 500 MB disk space for binaries, rules, and database
• Internet connectivity (optional, for email alerts)
• GCC compiler or pre-built binaries

Check system requirements:

uname -r
free -h
df -h /
gcc --version

Installation

Install OSSEC from source for maximum control and security. The installation process involves compiling the source code and configuring the installation.

Download OSSEC source code:

cd /tmp
wget https://github.com/ossec/ossec-hids/archive/3.7.0.tar.gz
tar xzf 3.7.0.tar.gz
cd ossec-hids-3.7.0

For CentOS/RHEL, install build tools:

sudo yum groupinstall -y 'Development Tools'
sudo yum install -y openssl-devel zlib-devel

For Ubuntu/Debian:

sudo apt-get install -y build-essential openssl libssl-dev zlib1g zlib1g-dev

OSSEC Architecture

OSSEC operates with two primary components:

1. Manager/Server: Central component that receives alerts from agents, processes rules, and stores data
2. Agent: Lightweight client installed on monitored servers that collects logs and file changes

A typical deployment includes one central manager receiving data from multiple distributed agents. For standalone deployments, the manager includes local_files for monitoring localhost.

Server Installation

Install the OSSEC manager on a central server that will receive and aggregate alerts from all agents.

Run the installation script:

cd /tmp/ossec-hids-3.7.0
sudo ./install.sh

The installer will prompt for configuration. Select the following options:

• Language: 1 (English)
• Installation type: Server
• Installation path: /var/ossec (default)
• Configuration method: Automatic

Or use automated installation:

sudo OSSEC_INSTALL_TYPE="server" ./install.sh auto

After installation, verify the installation:

ls -la /var/ossec/
/var/ossec/bin/wazuh-control start
/var/ossec/bin/wazuh-control status

Start the OSSEC service:

sudo systemctl start ossec
sudo systemctl enable ossec

Access the OSSEC main configuration file:

sudo nano /var/ossec/etc/ossec.conf

Key configuration sections:

<!-- OSSEC Configuration -->
<ossec_config>
  <global>
    <email_notification>yes</email_notification>
    <email_from>[email protected]</email_from>
    <smtp_server>localhost</smtp_server>
    <email_log_source>alerts.log</email_log_source>
    <white_list>127.0.0.1</white_list>
    <white_list>::1</white_list>
  </global>

  <rules>
    <include>default</include>
    <include>custom_rules</include>
  </rules>

  <logging>
    <log_alert_level>3</log_alert_level>
    <log_format>json</log_format>
  </logging>
</ossec_config>

Agent Installation

Install OSSEC agents on servers you want to monitor. Agents send data to the central manager.

Download the agent package:

cd /tmp
wget https://github.com/ossec/ossec-hids/archive/3.7.0.tar.gz
tar xzf 3.7.0.tar.gz
cd ossec-hids-3.7.0

Run the installer in agent mode:

sudo ./install.sh

Select these options:

• Language: 1 (English)
• Installation type: Agent
• Manager server IP: Enter the manager's IP address
• Agent name: Enter a descriptive name

Automated agent installation:

sudo OSSEC_INSTALL_TYPE="agent" OSSEC_MANAGER_IP="192.168.1.100" ./install.sh auto

Configure agent monitoring on the agent machine:

sudo nano /var/ossec/etc/ossec.conf

Add directories to monitor:

<agent_config>
  <localfile>
    <log_format>syslog</log_format>
    <location>/var/log/auth.log</location>
  </localfile>

  <localfile>
    <log_format>syslog</log_format>
    <location>/var/log/syslog</location>
  </localfile>

  <localfile>
    <log_format>syslog</log_format>
    <location>/var/log/apache2/error.log</location>
  </localfile>

  <localfile>
    <log_format>json</log_format>
    <location>/var/log/application/app.log</location>
  </localfile>

  <!-- Monitor SSH attempts -->
  <localfile>
    <log_format>syslog</log_format>
    <location>/var/log/auth.log</location>
  </localfile>

  <!-- Monitor web server access -->
  <localfile>
    <log_format>apache</log_format>
    <location>/var/log/apache2/access.log</location>
  </localfile>
</agent_config>

Start the agent:

sudo /var/ossec/bin/wazuh-control start

Verify agent status on the manager:

/var/ossec/bin/agent_control -lc
/var/ossec/bin/agent_control -i 001

List all connected agents:

/var/ossec/bin/agent_control -l

Show specific agent status:

/var/ossec/bin/agent_control -i 001 -s

Rule Creation and Management

OSSEC rules define how to respond to specific log entries. Rules are written in XML and include patterns, thresholds, and actions.

View existing rules:

ls -la /var/ossec/rules/

Create custom rules:

sudo nano /var/ossec/etc/rules/custom_rules.xml

Example custom rules:

<!-- OSSEC Custom Rules -->

<!-- Detect SSH brute force attempts -->
<group name="sshd">
  <rule id="100001" level="3">
    <if_sid>5710</if_sid>
    <match>Invalid user</match>
    <description>Invalid SSH login attempt</description>
  </rule>

  <rule id="100002" level="5">
    <if_sid>5710</if_sid>
    <match>^Invalid user</match>
    <options>alert_by_email</options>
    <description>Multiple invalid SSH login attempts</description>
  </rule>

  <!-- SSH brute force detection using threshold -->
  <rule id="100003" level="6">
    <if_sid>5710</if_sid>
    <frequency>5</frequency>
    <timeframe>120</timeframe>
    <options>alert_by_email</options>
    <description>SSH Brute Force Attack Detected</description>
  </rule>
</group>

<!-- Detect failed sudo attempts -->
<group name="sudo">
  <rule id="100004" level="4">
    <match>sudo:.*COMMAND</match>
    <description>Sudo command executed</description>
  </rule>

  <rule id="100005" level="5">
    <match>sudo:.*illegal user</match>
    <description>Sudo executed by unauthorized user</description>
    <options>alert_by_email</options>
  </rule>
</group>

<!-- Detect file modifications -->
<group name="syscheck">
  <rule id="100006" level="7">
    <if_sid>550</if_sid>
    <description>Critical file changed: /etc/passwd</description>
    <options>alert_by_email</options>
  </rule>

  <rule id="100007" level="7">
    <if_sid>550</if_sid>
    <description>Critical file changed: /etc/shadow</description>
    <options>alert_by_email</options>
  </rule>

  <rule id="100008" level="5">
    <if_sid>550</if_sid>
    <description>System configuration file changed</description>
  </rule>
</group>

<!-- Detect rootkit activity -->
<group name="rootcheck">
  <rule id="100009" level="8">
    <if_sid>510</if_sid>
    <description>Possible rootkit detected</description>
    <options>alert_by_email</options>
  </rule>
</group>

<!-- Detect port scanning -->
<group name="network">
  <rule id="100010" level="6">
    <match>Connection attempt</match>
    <frequency>10</frequency>
    <timeframe>60</timeframe>
    <description>Possible port scanning detected</description>
    <options>alert_by_email</options>
  </rule>
</group>

<!-- Detect privilege escalation -->
<group name="security">
  <rule id="100011" level="7">
    <match>sudo.*NOPASSWD</match>
    <description>Sudo NOPASSWD configuration detected</description>
    <options>alert_by_email</options>
  </rule>

  <rule id="100012" level="8">
    <match>setuid</match>
    <description>Setuid binary creation detected</description>
    <options>alert_by_email</options>
  </rule>
</group>

Verify rule syntax:

/var/ossec/bin/wazuh-control restart
tail -f /var/ossec/logs/ossec.log

Test rules with sample logs:

/var/ossec/bin/ossec-makelists
/var/ossec/bin/wazuh-control restart

Syscheck Configuration

Syscheck monitors file integrity and detects unauthorized modifications. This is crucial for detecting system compromise and rootkit installations.

Configure syscheck on the agent:

sudo nano /var/ossec/etc/ossec.conf

Add syscheck configuration:

<ossec_config>
  <syscheck>
    <!-- Check every 3600 seconds (1 hour) -->
    <frequency>3600</frequency>
    
    <!-- Check ownership and permissions -->
    <directories check_all="yes" realtime="yes">/etc</directories>
    <directories check_all="yes" realtime="yes">/usr/bin</directories>
    <directories check_all="yes" realtime="yes">/usr/sbin</directories>
    <directories check_all="yes" realtime="yes">/bin</directories>
    <directories check_all="yes" realtime="yes">/sbin</directories>
    <directories check_all="yes" realtime="yes">/root/.ssh</directories>
    <directories check_all="yes" realtime="yes">/root/.bash_history</directories>
    <directories check_all="yes" realtime="yes">/root/.bashrc</directories>
    
    <!-- Check web directories -->
    <directories check_all="yes" realtime="yes">/var/www/html</directories>
    
    <!-- Check cron directories -->
    <directories check_all="yes" realtime="yes">/etc/cron.d</directories>
    <directories check_all="yes" realtime="yes">/var/spool/cron</directories>
    
    <!-- Hash checks: OSSEC will compute and verify file hashes -->
    <directories check_all="yes" check_md5sum="yes" check_sha1sum="yes" realtime="yes">/etc/services</directories>
    
    <!-- Ignore certain files to reduce noise -->
    <ignore>/etc/mtab</ignore>
    <ignore>/etc/motd</ignore>
    <ignore>/etc/hosts.allow</ignore>
    <ignore>/etc/hosts.deny</ignore>
    <ignore>/proc</ignore>
    <ignore>/sys</ignore>
    
    <!-- Alert on specific file modifications -->
    <alert_new_files>yes</alert_new_files>
    <alert_changed_perms>yes</alert_changed_perms>
  </syscheck>
</ossec_config>

The realtime option enables real-time monitoring of file changes without waiting for the scheduled check interval.

Restart the agent to activate syscheck:

sudo /var/ossec/bin/wazuh-control restart

View syscheck alerts:

tail -f /var/ossec/logs/alerts/alerts.log | grep syscheck

Manually run syscheck:

/var/ossec/bin/syscheck_control -r

View syscheck database:

ls -la /var/ossec/queue/syscheck/

Rootcheck Setup

Rootcheck detects rootkit installations and other suspicious system modifications. This is critical for detecting sophisticated attacks.

Configure rootcheck:

sudo nano /var/ossec/etc/ossec.conf

Add rootcheck configuration:

<ossec_config>
  <rootcheck>
    <!-- Check every 36000 seconds (10 hours) -->
    <frequency>36000</frequency>
    
    <!-- Enable all rootkit checks -->
    <rootkit_files>/var/ossec/etc/shared/rootkit_files.txt</rootkit_files>
    <rootkit_trojans>/var/ossec/etc/shared/rootkit_trojans.txt</rootkit_trojans>
    
    <!-- Check system binaries for rootkits -->
    <skip_nfs>yes</skip_nfs>
    
    <!-- Enable detailed logging -->
    <verbose>yes</verbose>
    
    <!-- Check for suspicious processes -->
    <check_files>yes</check_files>
    <check_ports>yes</check_ports>
    <check_if>yes</check_if>
    
    <!-- System calls monitoring -->
    <check_sys_calls>yes</check_sys_calls>
  </rootcheck>
</ossec_config>

Restart to activate rootcheck:

sudo /var/ossec/bin/wazuh-control restart

Monitor rootcheck alerts:

tail -f /var/ossec/logs/alerts/alerts.log | grep rootcheck

Manually run rootcheck:

/var/ossec/bin/rootcheck_control -r

Review rootkit detection database:

cat /var/ossec/etc/shared/rootkit_files.txt | head -20
cat /var/ossec/etc/shared/rootkit_trojans.txt | head -20

Active Response

Active response triggers automated actions in response to security events. These actions can block IPs, kill processes, or execute custom scripts.

Configure active response on the manager:

sudo nano /var/ossec/etc/ossec.conf

Add active response configuration:

<ossec_config>
  <!-- Define an active response script -->
  <command>
    <name>restart-apache</name>
    <executable>restart-apache.sh</executable>
    <timeout_allowed>yes</timeout_allowed>
  </command>

  <!-- IP blocking active response -->
  <command>
    <name>firewall-drop</name>
    <executable>firewall-drop.sh</executable>
    <timeout_allowed>yes</timeout_allowed>
  </command>

  <!-- Define when to trigger active responses -->
  <active-response>
    <command>firewall-drop</command>
    <location>local</location>
    <rules_id>100003</rules_id>
    <timeout>600</timeout>
  </active-response>

  <!-- Restart Apache on web server attacks -->
  <active-response>
    <command>restart-apache</command>
    <location>server</location>
    <rules_id>100001</rules_id>
    <timeout>300</timeout>
  </active-response>

  <!-- Kill malicious process -->
  <command>
    <name>kill-process</name>
    <executable>kill-process.sh</executable>
    <timeout_allowed>yes</timeout_allowed>
  </command>

  <active-response>
    <command>kill-process</command>
    <location>agent</location>
    <rules_id>100012</rules_id>
    <timeout>0</timeout>
  </active-response>
</ossec_config>

Create a custom active response script:

sudo nano /var/ossec/active-response/bin/restart-apache.sh

Example script:

#!/bin/bash
ATTACKER_IP=$1
HOSTNAME=$2
ACTION=$3
ID=$4

if [ "$ACTION" = "add" ]; then
  systemctl restart apache2
  echo "Apache restarted due to security alert from $ATTACKER_IP" >> /var/log/ossec-response.log
fi

Make the script executable:

sudo chmod +x /var/ossec/active-response/bin/restart-apache.sh

Restart OSSEC:

sudo /var/ossec/bin/wazuh-control restart

Test active response:

# Simulate an attack alert
/var/ossec/bin/agent_control -i 001 -p

Email Alerts

Configure email notifications for critical security events. This ensures immediate notification of threats.

Install a mail server (if not present):

sudo apt-get install -y postfix
sudo dpkg-reconfigure postfix

Configure email in OSSEC:

sudo nano /var/ossec/etc/ossec.conf

Update the global section:

<global>
  <email_notification>yes</email_notification>
  <email_from>[email protected]</email_from>
  <smtp_server>smtp.yourdomain.com</smtp_server>
  <email_maxperhour>10</email_maxperhour>
  <email_idsname>your-server-name</email_idsname>
  
  <!-- Alert grouping (send alerts in batches) -->
  <log_alert_level>3</log_alert_level>
  <white_list>127.0.0.1</white_list>
  <white_list>::1</white_list>
</global>

Define email recipients for specific alerts:

<email_notification>
  <email_to>[email protected]</email_to>
  <level>7</level>
  <event_location>agent-name</event_location>
</email_notification>

<email_notification>
  <email_to>[email protected]</email_to>
  <level>5</level>
  <group>syscheck</group>
</email_notification>

Restart OSSEC:

sudo /var/ossec/bin/wazuh-control restart

Test email notifications:

echo "This is a test email" | mail -s "OSSEC Test" [email protected]

View email alerts that were sent:

grep "Sending email notification" /var/ossec/logs/ossec.log | tail -20

Log Aggregation

Aggregate logs from multiple agents for centralized analysis and archival.

Configure remote syslog reception on the manager:

sudo nano /var/ossec/etc/ossec.conf

Add syslog input:

<ossec_config>
  <remote>
    <connection>syslog</connection>
    <port>514</port>
    <protocol>udp</protocol>
  </remote>
</ossec_config>

Configure agents to send logs to syslog:

sudo nano /var/ossec/etc/ossec.conf

Add on each agent:

<ossec_config>
  <syslog_output>
    <server>192.168.1.100</server>
    <port>514</port>
  </syslog_output>
</ossec_config>

Verify agent connectivity to manager:

/var/ossec/bin/agent_control -i 001 -s

Review aggregated logs:

tail -f /var/ossec/logs/alerts/alerts.log
jq '.' /var/ossec/logs/alerts/alerts.json | head -50

Archive old logs:

tar czf /archive/ossec-logs-$(date +%Y%m%d).tar.gz /var/ossec/logs/archives/
find /var/ossec/logs/archives/ -mtime +90 -delete

Conclusion

OSSEC provides comprehensive host-based intrusion detection across your entire infrastructure. By following this guide, you've installed and configured the OSSEC manager for centralized alert collection, deployed agents on monitored servers, created custom detection rules for your environment, implemented syscheck for file integrity monitoring and rootcheck for rootkit detection, configured active response for automated threat mitigation, and set up email alerts for critical security events. Regular rule tuning, log review, and agent maintenance ensure sustained security posture. Whether protecting small deployments or large enterprise networks, OSSEC detects and responds to threats at the host level where attacks ultimately succeed or fail.