OSSEC is an open source host-based intrusion detection system (HIDS). Instead of watching network traffic, it runs on the server itself: it analyzes system logs, checks important files for unexpected changes (file integrity monitoring), looks for rootkits and can react to attacks automatically, for example by blocking an IP address that is brute forcing SSH. In this tutorial you will compile and install OSSEC in server mode on Ubuntu 24.04, write and test a custom rule, configure real-time file integrity monitoring, enable automatic blocking and connect a second server as an agent.
Prerequisites
To follow this tutorial you need:
- A server running Ubuntu 24.04 LTS, for example a CubePath VPS, with at least 1 GB of RAM. This will be the OSSEC server (manager).
- A non-root user with
sudoprivileges. - The public IP address of the machine you connect from, so you can whitelist it before enabling automatic blocking.
- Optionally, a second Ubuntu 24.04 server to install as an agent in Step 7.
Step 1 - Installing build dependencies
OSSEC is distributed as source code and compiled during installation. Install the compiler and the libraries it links against:
sudo apt update
sudo apt install build-essential make curl zlib1g-dev libpcre2-dev libevent-dev libssl-dev libsystemd-dev
libsystemd-dev lets OSSEC read the systemd journal, and libpcre2-dev provides the regular expression library used by the rules.
Step 2 - Downloading and installing OSSEC
Download the latest release from the official GitHub repository. At the time of writing this is 3.7.0; check the releases page and adjust the version if a newer one exists:
cd ~
curl -fLO https://github.com/ossec/ossec-hids/archive/3.7.0.tar.gz
tar -xzf 3.7.0.tar.gz
cd ossec-hids-3.7.0
Run the installer. PCRE2_SYSTEM=yes tells the build to use the PCRE2 library you just installed instead of downloading its own copy:
sudo PCRE2_SYSTEM=yes ./install.sh
The installer asks a series of questions. Answer them as follows:
(en/br/cn/de/el/es/fr/hu/it/jp/nl/pl/ru/sr/tr) [en]: en
1- What kind of installation do you want (server, agent, local, hybrid or help)? server
2- Choose where to install the OSSEC HIDS [/var/ossec]: (press Enter)
3.1- Do you want e-mail notification? (y/n) [y]: n
3.2- Do you want to run the integrity check daemon? (y/n) [y]: y
3.3- Do you want to run the rootkit detection engine? (y/n) [y]: y
3.4- Do you want to enable active response? (y/n) [y]: y
- Do you want to enable the firewall-drop response? (y/n) [y]: y
- Do you want to add more IPs to the white list? (y/n)? [n]: y
- IPs (space separated): your_admin_ip
3.5- Do you want to enable remote syslog (port 514 udp)? (y/n) [y]: n
Replace your_admin_ip with the public IP you connect from. Whitelisted addresses are never blocked by active response, which protects you from locking yourself out. Choose server even for a single machine: the server monitors itself and can accept agents later. E-mail is disabled here because it requires a working mail relay; you can enable it later in ossec.conf.
The compilation takes a couple of minutes and ends with:
- Configuration finished properly.
...
--- Press ENTER to finish (maybe more information below). ---
Step 3 - Starting OSSEC and checking its processes
Start OSSEC with its control script:
sudo /var/ossec/bin/ossec-control start
Check that all daemons are running:
sudo /var/ossec/bin/ossec-control status
ossec-monitord is running...
ossec-logcollector is running...
ossec-remoted is running...
ossec-syscheckd is running...
ossec-analysisd is running...
ossec-maild not running...
ossec-execd is running...
ossec-maild is stopped because e-mail notification is disabled. The installer also registers OSSEC to start at boot.
The two files you will use most are the main log and the alert log:
sudo tail -n 20 /var/ossec/logs/ossec.log
sudo tail -f /var/ossec/logs/alerts/alerts.log
Press Ctrl+C to stop following the alert log.
Ubuntu 24.04 servers still write authentication events to /var/log/auth.log through rsyslog, which OSSEC monitors by default. Confirm the file exists and is being written to:
sudo tail -n 3 /var/log/auth.log
If it does not exist, install rsyslog with sudo apt install rsyslog and restart OSSEC.
Step 4 - Writing and testing a custom rule
OSSEC decodes each log line and matches it against XML rules in /var/ossec/rules/. Do not edit the bundled rule files, as upgrades overwrite them. Put your own rules in local_rules.xml, using IDs between 100000 and 119999.
Before writing a rule, see how OSSEC classifies a sample event. ossec-logtest reads log lines from standard input and shows the decoder and the matching rule:
sudo /var/ossec/bin/ossec-logtest
Paste this line and press Enter:
Sep 25 10:00:00 web1 sshd[1234]: Failed password for invalid user admin from 203.0.113.50 port 51234 ssh2
**Phase 1: Completed pre-decoding.
full event: 'Sep 25 10:00:00 web1 sshd[1234]: Failed password for invalid user admin from 203.0.113.50 port 51234 ssh2'
hostname: 'web1'
program_name: 'sshd'
log: 'Failed password for invalid user admin from 203.0.113.50 port 51234 ssh2'
**Phase 2: Completed decoding.
decoder: 'sshd'
srcip: '203.0.113.50'
**Phase 3: Completed filtering (rules).
Rule id: '5710'
Level: '5'
Description: 'Attempt to login using a non-existent user'
**Alert to be generated.
Press Ctrl+C to exit. Rule 5710 has level 5. Suppose you want login attempts for well-known default accounts to stand out with a higher level. Open the local rules file:
sudo nano /var/ossec/rules/local_rules.xml
Add this group at the end of the file:
<group name="local,sshd,">
<rule id="100100" level="10">
<if_sid>5710</if_sid>
<match>invalid user admin |invalid user test |invalid user oracle </match>
<description>SSH login attempt with a common default username</description>
<group>authentication_failed,</group>
</rule>
</group>
if_sid makes the rule a child of 5710, so it is only evaluated when 5710 matches, and match accepts several patterns separated by |. Run ossec-logtest again with the same line. Phase 3 should now report your rule:
**Phase 3: Completed filtering (rules).
Rule id: '100100'
Level: '10'
Description: 'SSH login attempt with a common default username'
**Alert to be generated.
Restart OSSEC so the running analysis daemon loads the new rule:
sudo /var/ossec/bin/ossec-control restart
Step 5 - Configuring real-time file integrity monitoring
Syscheck, the integrity checker, stores checksums, permissions and ownership of monitored files and alerts when they change. By default it scans every 22 hours. For critical directories you can enable real-time monitoring, which uses inotify and reports changes within seconds.
Open the main configuration file:
sudo nano /var/ossec/etc/ossec.conf
Find the <syscheck> section and replace the default <directories> lines with the following. Real-time monitoring only works on directories, not individual files:
<syscheck>
<frequency>43200</frequency>
<alert_new_files>yes</alert_new_files>
<directories check_all="yes" realtime="yes">/etc,/usr/bin,/usr/sbin</directories>
<directories check_all="yes">/bin,/sbin,/boot</directories>
<directories check_all="yes" realtime="yes">/root/.ssh</directories>
<ignore>/etc/mtab</ignore>
<ignore>/etc/hosts.deny</ignore>
<ignore>/etc/random-seed</ignore>
<ignore>/etc/adjtime</ignore>
</syscheck>
Leave any other options that were already in the section. frequency sets the full scan interval to 12 hours and alert_new_files reports files that appear in monitored directories. If you host websites, add their document root (for example /var/www) with realtime="yes" to detect injected web shells.
Restart OSSEC:
sudo /var/ossec/bin/ossec-control restart
The first scan builds the baseline and does not generate alerts. Wait for it to complete:
sudo grep -i "syscheck" /var/ossec/logs/ossec.log | tail -n 3
2026/09/25 10:42:10 ossec-syscheckd: INFO: Starting syscheck scan (forwarding database).
2026/09/25 10:44:37 ossec-syscheckd: INFO: Ending syscheck scan (forwarding database).
Now change a monitored file. Adding a comment to /etc/hosts is harmless:
echo "# ossec test" | sudo tee -a /etc/hosts
Within a few seconds an alert appears:
sudo grep -A 6 "Integrity checksum changed" /var/ossec/logs/alerts/alerts.log | tail -n 7
Rule: 550 (level 7) -> 'Integrity checksum changed.'
Integrity checksum changed for: '/etc/hosts'
Size changed from '221' to '234'
Old md5sum was: '3f1c...'
New md5sum was: '9a0b...'
Old sha1sum was: '1c2d...'
New sha1sum was: '77e4...'
Remove the test line afterwards with sudo nano /etc/hosts.
Step 6 - Blocking brute force attacks with active response
Active response runs a script when a rule matches. You enabled the firewall-drop response during installation; it inserts a firewall rule that drops traffic from the attacking IP and removes it after a timeout. Open /var/ossec/etc/ossec.conf with sudo nano and find the <active-response> block that the installer added (it follows a comment about blocking the IP for 600 seconds). It looks like this:
<active-response>
<command>firewall-drop</command>
<location>local</location>
<level>6</level>
<timeout>600</timeout>
</active-response>
Every alert of level 6 or higher that carries a source IP blocks that IP for 600 seconds. That includes rule 5712 (SSH brute force, several failed logins in a short time) and the custom rule 100100 you wrote in Step 4. Level 6 is aggressive; if you see legitimate users blocked, raise it to 10 so only brute force patterns trigger it.
Before relying on it, confirm your own address is whitelisted in the <global> section:
sudo grep "white_list" /var/ossec/etc/ossec.conf
<white_list>127.0.0.1</white_list>
<white_list>^localhost.localdomain$</white_list>
<white_list>your_admin_ip</white_list>
To test it, try to log in several times with a wrong user from a machine that is not whitelisted, for example ssh admin@your_server_ip. Then check the active response log on the server:
sudo tail -n 2 /var/ossec/logs/active-responses.log
Thu Sep 25 10:51:03 UTC 2026 /var/ossec/active-response/bin/firewall-drop.sh add - 198.51.100.7 1758797463.44810 100100
The blocked address is visible in the firewall:
sudo iptables -S INPUT | grep 198.51.100.7
-A INPUT -s 198.51.100.7/32 -j DROP
After 600 seconds OSSEC runs the same script with delete and the rule disappears.
Step 7 - (Optional) Adding an agent
To monitor more servers from one place, install OSSEC in agent mode on each of them. Agents send events to the server on UDP port 1514, where they are analyzed and alerted centrally.
On the OSSEC server, allow the agent's IP address through UFW if the firewall is active:
sudo ufw allow from agent_ip to any port 1514 proto udp
Register the agent on the server with manage_agents:
sudo /var/ossec/bin/manage_agents
Choose A to add an agent, enter a name (for example web2), the agent's IP address and accept the proposed ID 001. Then choose E, enter ID 001 and copy the long base64 key it prints. Choose Q to quit and restart the server so it accepts the new agent:
sudo /var/ossec/bin/ossec-control restart
On the agent machine, repeat Steps 1 and 2, but answer agent to the installation type and enter the server's IP address when asked for it. Then import the key:
sudo /var/ossec/bin/manage_agents
Choose I, paste the key and confirm with y, then Q to quit. Start the agent:
sudo /var/ossec/bin/ossec-control start
Back on the server, list the agents and their status:
sudo /var/ossec/bin/agent_control -l
OSSEC HIDS agent_control. List of available agents:
ID: 000, Name: ossec-server (server), IP: 127.0.0.1, Active/Local
ID: 001, Name: web2, IP: 198.51.100.20, Active
Rules and active responses on the server now apply to events coming from the agent too.
Troubleshooting
- The build fails with an error about PCRE2. Make sure
libpcre2-devis installed and that you ran the installer withPCRE2_SYSTEM=yes. - An agent stays
Never connected. Check that UDP 1514 is open on the server, that the IP entered inmanage_agentsmatches the agent's real source address, and read/var/ossec/logs/ossec.logon both sides. - You blocked yourself. Log in through the VPS console and run
sudo iptables -D INPUT -s your_admin_ip -j DROP, then add your IP as a<white_list>entry in the<global>section and restart OSSEC. - Too many syscheck alerts. Add
<ignore>entries for files that change legitimately, such as caches or generated configuration.
Conclusion
OSSEC is now analyzing your server's logs, watching critical directories in real time, blocking brute force sources automatically and, optionally, collecting events from agents. Review /var/ossec/logs/alerts/alerts.log regularly during the first days and tune rules and ignores. As next steps, enable e-mail alerts with a local relay such as Postfix, forward alerts to your log platform, or evaluate Wazuh, an actively developed fork of OSSEC with a web dashboard.