OSSEC Host-Based Intrusion Detection

OSSEC is a comprehensive host-based intrusion detection system that proporciona real-time monitoring of file integrity, system logs, rootkits, and active response capabilities. Unlike red-based IDS systems, OSSEC operates on individual servers and endpoints, detecting attacks from internal compromise, privilege escalation, and system tampering. Esta guía cubre installing OSSEC in server and agent configurations, creating and managing detection rules, configuring syscheck for file integrity monitoring, implementing active response mechanisms, and setting up email alerts for critical security events.

Tabla de Contenidos

• System Requirements
• Instalación
• OSSEC Architecture
• Server Instalación
• Agent Instalación
• Rule Creation and Gestión
• Syscheck Configuración
• Rootcheck Configuración
• Active Response
• Email Alerts
• Log Aggregation
• Conclusión

System Requirements

OSSEC requires minimal system resources and can run on older hardware. Ensure these prerequisites:

• Linux kernel 2.6 or newer
• At least 512 MB RAM (1 GB+ for large deployments)
• 500 MB disk space for binaries, rules, and database
• Internet connectivity (optional, for email alerts)
• GCC compiler or pre-built binaries

Check system requirements:

uname -r
free -h
df -h /
gcc --version

Instalación

Instala OSSEC from source for maximum control and security. The installation process involves compiling the source code and configuring the installation.

Download OSSEC source code:

cd /tmp
wget https://github.com/ossec/ossec-hids/archive/3.7.0.tar.gz
tar xzf 3.7.0.tar.gz
cd ossec-hids-3.7.0

For CentOS/RHEL, install build tools:

sudo yum groupinstall -y 'Development Tools'
sudo yum install -y openssl-devel zlib-devel

For Ubuntu/Debian:

sudo apt-get install -y build-essential openssl libssl-dev zlib1g zlib1g-dev

OSSEC Architecture

OSSEC operates with two primary components:

1. Manager/Server: Central component that receives alerts from agents, processes rules, and stores data
2. Agent: Lightweight client installed on monitored servers that collects logs and file changes

A typical deployment includes one central manager receiving data from multiple distributed agents. For standalone deployments, the manager includes local_files for monitoring localhost.

Server Instalación

Instala the OSSEC manager on a central server that will receive and aggregate alerts from all agents.

Ejecuta the installation script:

cd /tmp/ossec-hids-3.7.0
sudo ./install.sh

The installer will prompt for configuration. Select the following options:

• Language: 1 (English)
• Instalación type: Server
• Instalación path: /var/ossec (default)
• Configuración method: Automatic

Or use automated installation:

sudo OSSEC_INSTALL_TYPE="server" ./install.sh auto

After installation, verifica the installation:

ls -la /var/ossec/
/var/ossec/bin/wazuh-control start
/var/ossec/bin/wazuh-control status

Inicia the OSSEC servicio:

sudo systemctl start ossec
sudo systemctl enable ossec

Access the OSSEC main configuration file:

sudo nano /var/ossec/etc/ossec.conf

Key configuration sections:

<!-- OSSEC Configuración -->
<ossec_config>
  <global>
    <email_notification>yes</email_notification>
    <email_from>[email protected]</email_from>
    <smtp_server>localhost</smtp_server>
    <email_log_source>alerts.log</email_log_source>
    <white_list>127.0.0.1</white_list>
    <white_list>::1</white_list>
  </global>

  <rules>
    <include>default</include>
    <include>custom_rules</include>
  </rules>

  <logging>
    <log_alert_level>3</log_alert_level>
    <log_format>json</log_format>
  </logging>
</ossec_config>

Agent Instalación

Instala OSSEC agents on servers you want to monitor. Agents send data to the central manager.

Download the agent package:

cd /tmp
wget https://github.com/ossec/ossec-hids/archive/3.7.0.tar.gz
tar xzf 3.7.0.tar.gz
cd ossec-hids-3.7.0

Ejecuta the installer in agent mode:

sudo ./install.sh

Select these options:

• Language: 1 (English)
• Instalación type: Agent
• Manager server IP: Enter the manager's IP address
• Agent name: Enter a descriptive name

Automated agent installation:

sudo OSSEC_INSTALL_TYPE="agent" OSSEC_MANAGER_IP="192.168.1.100" ./install.sh auto

Configura agent monitoring on the agent machine:

sudo nano /var/ossec/etc/ossec.conf

Add directories to monitor:

<agent_config>
  <localfile>
    <log_format>syslog</log_format>
    <location>/var/log/auth.log</location>
  </localfile>

  <localfile>
    <log_format>syslog</log_format>
    <location>/var/log/syslog</location>
  </localfile>

  <localfile>
    <log_format>syslog</log_format>
    <location>/var/log/apache2/error.log</location>
  </localfile>

  <localfile>
    <log_format>json</log_format>
    <location>/var/log/application/app.log</location>
  </localfile>

  <!-- Monitorea SSH attempts -->
  <localfile>
    <log_format>syslog</log_format>
    <location>/var/log/auth.log</location>
  </localfile>

  <!-- Monitorea web server access -->
  <localfile>
    <log_format>apache</log_format>
    <location>/var/log/apache2/access.log</location>
  </localfile>
</agent_config>

Inicia the agent:

sudo /var/ossec/bin/wazuh-control start

Verifica agent status on the manager:

/var/ossec/bin/agent_control -lc
/var/ossec/bin/agent_control -i 001

List all connected agents:

/var/ossec/bin/agent_control -l

Show specific agent status:

/var/ossec/bin/agent_control -i 001 -s

Rule Creation and Gestión

OSSEC rules define how to respond to specific log entries. Rules are written in XML and include patterns, thresholds, and actions.

View existing rules:

ls -la /var/ossec/rules/

Crea custom rules:

sudo nano /var/ossec/etc/rules/custom_rules.xml

Example custom rules:

<!-- OSSEC Custom Rules -->

<!-- Detect SSH brute force attempts -->
<group name="sshd">
  <rule id="100001" level="3">
    <if_sid>5710</if_sid>
    <match>Invalid user</match>
    <description>Invalid SSH login attempt</description>
  </rule>

  <rule id="100002" level="5">
    <if_sid>5710</if_sid>
    <match>^Invalid user</match>
    <options>alert_by_email</options>
    <description>Multiple invalid SSH login attempts</description>
  </rule>

  <!-- SSH brute force detection using threshold -->
  <rule id="100003" level="6">
    <if_sid>5710</if_sid>
    <frequency>5</frequency>
    <timeframe>120</timeframe>
    <options>alert_by_email</options>
    <description>SSH Brute Force Attack Detected</description>
  </rule>
</group>

<!-- Detect failed sudo attempts -->
<group name="sudo">
  <rule id="100004" level="4">
    <match>sudo:.*COMMAND</match>
    <description>Sudo command executed</description>
  </rule>

  <rule id="100005" level="5">
    <match>sudo:.*illegal user</match>
    <description>Sudo executed by unauthorized user</description>
    <options>alert_by_email</options>
  </rule>
</group>

<!-- Detect file modifications -->
<group name="syscheck">
  <rule id="100006" level="7">
    <if_sid>550</if_sid>
    <description>Critical file changed: /etc/passwd</description>
    <options>alert_by_email</options>
  </rule>

  <rule id="100007" level="7">
    <if_sid>550</if_sid>
    <description>Critical file changed: /etc/shadow</description>
    <options>alert_by_email</options>
  </rule>

  <rule id="100008" level="5">
    <if_sid>550</if_sid>
    <description>System configuration file changed</description>
  </rule>
</group>

<!-- Detect rootkit activity -->
<group name="rootcheck">
  <rule id="100009" level="8">
    <if_sid>510</if_sid>
    <description>Possible rootkit detected</description>
    <options>alert_by_email</options>
  </rule>
</group>

<!-- Detect puerto scanning -->
<group name="red">
  <rule id="100010" level="6">
    <match>Connection attempt</match>
    <frequency>10</frequency>
    <timeframe>60</timeframe>
    <description>Possible puerto scanning detected</description>
    <options>alert_by_email</options>
  </rule>
</group>

<!-- Detect privilege escalation -->
<group name="security">
  <rule id="100011" level="7">
    <match>sudo.*NOPASSWD</match>
    <description>Sudo NOPASSWD configuration detected</description>
    <options>alert_by_email</options>
  </rule>

  <rule id="100012" level="8">
    <match>setuid</match>
    <description>Setuid binary creation detected</description>
    <options>alert_by_email</options>
  </rule>
</group>

Verifica rule syntax:

/var/ossec/bin/wazuh-control restart
tail -f /var/ossec/logs/ossec.log

Prueba rules with sample logs:

/var/ossec/bin/ossec-makelists
/var/ossec/bin/wazuh-control restart

Syscheck Configuración

Syscheck monitors file integrity and detects unauthorized modifications. This is crucial for detecting system compromise and rootkit installations.

Configura syscheck on the agent:

sudo nano /var/ossec/etc/ossec.conf

Add syscheck configuration:

<ossec_config>
  <syscheck>
    <!-- Check every 3600 seconds (1 hour) -->
    <frequency>3600</frequency>
    
    <!-- Check ownership and permissions -->
    <directories check_all="yes" realtime="yes">/etc</directories>
    <directories check_all="yes" realtime="yes">/usr/bin</directories>
    <directories check_all="yes" realtime="yes">/usr/sbin</directories>
    <directories check_all="yes" realtime="yes">/bin</directories>
    <directories check_all="yes" realtime="yes">/sbin</directories>
    <directories check_all="yes" realtime="yes">/root/.ssh</directories>
    <directories check_all="yes" realtime="yes">/root/.bash_history</directories>
    <directories check_all="yes" realtime="yes">/root/.bashrc</directories>
    
    <!-- Check web directories -->
    <directories check_all="yes" realtime="yes">/var/www/html</directories>
    
    <!-- Check cron directories -->
    <directories check_all="yes" realtime="yes">/etc/cron.d</directories>
    <directories check_all="yes" realtime="yes">/var/spool/cron</directories>
    
    <!-- Hash checks: OSSEC will compute and verifica file hashes -->
    <directories check_all="yes" check_md5sum="yes" check_sha1sum="yes" realtime="yes">/etc/servicios</directories>
    
    <!-- Ignore certain files to reduce noise -->
    <ignore>/etc/mtab</ignore>
    <ignore>/etc/motd</ignore>
    <ignore>/etc/hosts.allow</ignore>
    <ignore>/etc/hosts.deny</ignore>
    <ignore>/proc</ignore>
    <ignore>/sys</ignore>
    
    <!-- Alert on specific file modifications -->
    <alert_new_files>yes</alert_new_files>
    <alert_changed_perms>yes</alert_changed_perms>
  </syscheck>
</ossec_config>

The realtime option enables real-time monitoring of file changes without waiting for the scheduled verifica interval.

Reinicia the agent to activate syscheck:

sudo /var/ossec/bin/wazuh-control restart

View syscheck alerts:

tail -f /var/ossec/logs/alerts/alerts.log | grep syscheck

Manually run syscheck:

/var/ossec/bin/syscheck_control -r

View syscheck database:

ls -la /var/ossec/queue/syscheck/

Rootcheck Configuración

Rootcheck detects rootkit installations and other suspicious system modifications. This is critical for detecting sophisticated attacks.

Configura rootcheck:

sudo nano /var/ossec/etc/ossec.conf

Add rootcheck configuration:

<ossec_config>
  <rootcheck>
    <!-- Check every 36000 seconds (10 hours) -->
    <frequency>36000</frequency>
    
    <!-- Habilita all rootkit checks -->
    <rootkit_files>/var/ossec/etc/shared/rootkit_files.txt</rootkit_files>
    <rootkit_trojans>/var/ossec/etc/shared/rootkit_trojans.txt</rootkit_trojans>
    
    <!-- Check system binaries for rootkits -->
    <skip_nfs>yes</skip_nfs>
    
    <!-- Habilita detailed logging -->
    <verbose>yes</verbose>
    
    <!-- Check for suspicious processes -->
    <check_files>yes</check_files>
    <check_ports>yes</check_ports>
    <check_if>yes</check_if>
    
    <!-- System calls monitoring -->
    <check_sys_calls>yes</check_sys_calls>
  </rootcheck>
</ossec_config>

Reinicia to activate rootcheck:

sudo /var/ossec/bin/wazuh-control restart

Monitorea rootcheck alerts:

tail -f /var/ossec/logs/alerts/alerts.log | grep rootcheck

Manually run rootcheck:

/var/ossec/bin/rootcheck_control -r

Review rootkit detection database:

cat /var/ossec/etc/shared/rootkit_files.txt | head -20
cat /var/ossec/etc/shared/rootkit_trojans.txt | head -20

Active Response

Active response triggers automated actions in response to security events. These actions can block IPs, kill processes, or execute custom scripts.

Configura active response on the manager:

sudo nano /var/ossec/etc/ossec.conf

Add active response configuration:

<ossec_config>
  <!-- Define an active response script -->
  <command>
    <name>restart-apache</name>
    <executable>restart-apache.sh</executable>
    <timeout_allowed>yes</timeout_allowed>
  </command>

  <!-- IP blocking active response -->
  <command>
    <name>firewall-drop</name>
    <executable>firewall-drop.sh</executable>
    <timeout_allowed>yes</timeout_allowed>
  </command>

  <!-- Define when to trigger active responses -->
  <active-response>
    <command>firewall-drop</command>
    <location>local</location>
    <rules_id>100003</rules_id>
    <timeout>600</timeout>
  </active-response>

  <!-- Reinicia Apache on web server attacks -->
  <active-response>
    <command>restart-apache</command>
    <location>server</location>
    <rules_id>100001</rules_id>
    <timeout>300</timeout>
  </active-response>

  <!-- Kill malicious process -->
  <command>
    <name>kill-process</name>
    <executable>kill-process.sh</executable>
    <timeout_allowed>yes</timeout_allowed>
  </command>

  <active-response>
    <command>kill-process</command>
    <location>agent</location>
    <rules_id>100012</rules_id>
    <timeout>0</timeout>
  </active-response>
</ossec_config>

Crea a custom active response script:

sudo nano /var/ossec/active-response/bin/restart-apache.sh

Example script:

#!/bin/bash
ATTACKER_IP=$1
HOSTNAME=$2
ACTION=$3
ID=$4

if [ "$ACTION" = "add" ]; then
  systemctl restart apache2
  echo "Apache restarted due to security alert from $ATTACKER_IP" >> /var/log/ossec-response.log
fi

Make the script executable:

sudo chmod +x /var/ossec/active-response/bin/restart-apache.sh

Reinicia OSSEC:

sudo /var/ossec/bin/wazuh-control restart

Prueba active response:

# Simulate an attack alert
/var/ossec/bin/agent_control -i 001 -p

Email Alerts

Configura email notifications for critical security events. This ensures immediate notification of threats.

Instala a mail server (if not present):

sudo apt-get install -y postfix
sudo dpkg-reconfigure postfix

Configura email in OSSEC:

sudo nano /var/ossec/etc/ossec.conf

Actualiza the global section:

<global>
  <email_notification>yes</email_notification>
  <email_from>[email protected]</email_from>
  <smtp_server>smtp.yourdomain.com</smtp_server>
  <email_maxperhour>10</email_maxperhour>
  <email_idsname>your-server-name</email_idsname>
  
  <!-- Alert grouping (send alerts in batches) -->
  <log_alert_level>3</log_alert_level>
  <white_list>127.0.0.1</white_list>
  <white_list>::1</white_list>
</global>

Define email recipients for specific alerts:

<email_notification>
  <email_to>[email protected]</email_to>
  <level>7</level>
  <event_location>agent-name</event_location>
</email_notification>

<email_notification>
  <email_to>[email protected]</email_to>
  <level>5</level>
  <group>syscheck</group>
</email_notification>

Reinicia OSSEC:

sudo /var/ossec/bin/wazuh-control restart

Prueba email notifications:

echo "This is a test email" | mail -s "OSSEC Prueba" [email protected]

View email alerts that were sent:

grep "Sending email notification" /var/ossec/logs/ossec.log | tail -20

Log Aggregation

Agrega logs from multiple agents for centralized analysis and archival.

Configura remote syslog reception on the manager:

sudo nano /var/ossec/etc/ossec.conf

Add syslog input:

<ossec_config>
  <remote>
    <connection>syslog</connection>
    <puerto>514</puerto>
    <protocol>udp</protocol>
  </remote>
</ossec_config>

Configura agents to send logs to syslog:

sudo nano /var/ossec/etc/ossec.conf

Add on each agent:

<ossec_config>
  <syslog_output>
    <server>192.168.1.100</server>
    <puerto>514</puerto>
  </syslog_output>
</ossec_config>

Verifica agent connectivity to manager:

/var/ossec/bin/agent_control -i 001 -s

Review aggregated logs:

tail -f /var/ossec/logs/alerts/alerts.log
jq '.' /var/ossec/logs/alerts/alerts.json | head -50

Archive old logs:

tar czf /archive/ossec-logs-$(date +%Y%m%d).tar.gz /var/ossec/logs/archives/
find /var/ossec/logs/archives/ -mtime +90 -delete

Conclusión

OSSEC proporciona comprehensive host-based intrusion detection across your entire infrastructure. By following this guide, you've installed and configured the OSSEC manager for centralized alert collection, deployed agents on monitored servers, created custom detection rules for your environment, implemented syscheck for file integrity monitoring and rootcheck for rootkit detection, configured active response for automated threat mitigation, and set up email alerts for critical security events. Regular rule tuning, log review, and agent maintenance asegúrate de que sustained security posture. Whether protecting small deployments or large enterprise redes, OSSEC detects and responds to threats at the host level where attacks ultimately succeed or fail.