--- title: "How to Analyze Linux Logs in /var/log and journalctl" description: "Find and read the right log files in /var/log and the systemd journal on Ubuntu 24.04, filter them by service, time and priority, and trace failed SSH logins." category: "Monitoring Logging" language: en source: https://cubepath.com/docs/monitoring-logging/linux-system-log-analysis-var-log updated: 2026-09-25 --- # How to Analyze Linux Logs in /var/log and journalctl When a service crashes, a disk fills up or someone tries to break into your server, the explanation is almost always in the logs. On Ubuntu 24.04, logs live in two places: plain text files under `/var/log`, written by rsyslog and by applications such as Nginx, and the binary systemd journal, read with `journalctl`. In this tutorial you will learn which file holds what, how to filter logs quickly by time, service and severity, and how to answer common questions such as "why did this service stop?" and "who is trying to log in over SSH?". ## Prerequisites To follow this guide you need: - A server running Ubuntu 24.04 LTS, for example a CubePath VPS. The commands also work on Debian 12, with the file name differences noted below. - A non-root user with `sudo` privileges. Most system logs are readable only by root and the `adm` group. On Rocky Linux and other RHEL-based systems, `/var/log/syslog` is called `/var/log/messages` and `/var/log/auth.log` is `/var/log/secure`. The `journalctl` commands are identical. ## Step 1 - Getting to know /var/log List the directory, sorted by modification time so the most active logs are at the top: ```bash ls -lt /var/log | head -n 15 ``` These are the files and directories you will use most on Ubuntu 24.04: | Path | Contents | |---|---| | `/var/log/syslog` | General system messages from services and the kernel | | `/var/log/auth.log` | Authentication: SSH logins, `sudo`, user and group changes | | `/var/log/kern.log` | Kernel messages: hardware, drivers, firewall, OOM killer | | `/var/log/dpkg.log` | Packages installed, upgraded and removed | | `/var/log/apt/history.log` | `apt` commands that were run and what they changed | | `/var/log/unattended-upgrades/` | Automatic security updates | | `/var/log/journal/` | The persistent systemd journal (binary, read with `journalctl`) | | `/var/log/nginx/`, `/var/log/apache2/`, `/var/log/mysql/` | Application logs, when those services are installed | Rotated logs sit next to the current file: `syslog.1` is the previous period and `syslog.2.gz`, `syslog.3.gz` are older compressed copies. > Note: On Debian 12, rsyslog is not installed by default, so `/var/log/syslog` and `/var/log/auth.log` do not exist. Use `journalctl` (Step 3) or install rsyslog with `sudo apt install rsyslog`. To read these files without typing `sudo` every time, you can add your user to the `adm` group and log in again: ```bash sudo usermod -aG adm your_user ``` ## Step 2 - Reading and searching text logs Show the last 50 lines of a log: ```bash sudo tail -n 50 /var/log/syslog ``` Follow a log live while you reproduce a problem, for example restarting a service in another terminal. Press `Ctrl+C` to stop: ```bash sudo tail -f /var/log/syslog ``` To page through a large file, use `less`. Press `G` to jump to the end, `/` to search forward, `n` for the next match, `F` to follow new lines like `tail -f` and `q` to quit: ```bash sudo less /var/log/syslog ``` Search for a term with `grep`. `-i` ignores case and `-C 2` shows two lines of context before and after each match: ```bash sudo grep -i -C 2 'error' /var/log/syslog ``` Count matches instead of printing them: ```bash sudo grep -ci 'error' /var/log/syslog ``` Rotated `.gz` files cannot be searched with `grep` directly. `zgrep` reads compressed and plain files alike, so you can search the whole history at once: ```bash sudo zgrep -h 'Out of memory' /var/log/kern.log* ``` ## Step 3 - Querying the systemd journal Every service managed by systemd sends its output to the journal, even when it writes no text log file. This makes `journalctl` the fastest way to see why a service failed. Check that the journal is persistent, so logs survive reboots. On Ubuntu 24.04 the directory exists by default: ```bash ls -d /var/log/journal ``` ```output /var/log/journal ``` Show the logs of one service. Service names are the unit names, for example `ssh`, `nginx` or `cron`: ```bash sudo journalctl -u ssh ``` The most useful filters, which you can combine: ```bash sudo journalctl -u nginx -n 100 --no-pager ``` Shows the last 100 lines of Nginx without the pager. ```bash sudo journalctl -u nginx -f ``` Follows new Nginx entries live. ```bash sudo journalctl --since "2026-09-25 09:00" --until "2026-09-25 10:00" ``` Limits output to a time window. Relative values like `--since "1 hour ago"` or `--since yesterday` also work. ```bash sudo journalctl -p err -b ``` Shows only messages of priority `err` or more severe (`crit`, `alert`, `emerg`) since the current boot. This is a good first command after an incident. ```bash sudo journalctl -k -b -1 ``` Shows kernel messages from the previous boot, which is where you find the reason for an unexpected reboot or a crash. List the boots the journal knows about: ```bash sudo journalctl --list-boots ``` ```output IDX BOOT ID FIRST ENTRY LAST ENTRY -1 5c1e0d3f2b7a4f3e9d7c8a1b2c3d4e5f Mon 2026-09-15 08:12:01 UTC Wed 2026-09-24 22:41:10 UTC 0 a9b8c7d6e5f4a3b2c1d0e9f8a7b6c5d4 Wed 2026-09-24 22:43:05 UTC Thu 2026-09-25 10:15:44 UTC ``` When you only need to know why a unit is failing, `systemctl status` shows its state together with the last log lines: ```bash sudo systemctl status nginx ``` ## Step 4 - Investigating SSH login attempts Any server with a public IP receives constant SSH brute-force attempts. `auth.log` records each one. Count failed password attempts per source IP: ```bash sudo grep 'Failed password' /var/log/auth.log | grep -oE 'from [0-9a-f.:]+' | awk '{print $2}' | sort | uniq -c | sort -rn | head ``` ```output 412 198.51.100.77 288 203.0.113.201 36 192.0.2.14 ``` See the attempts that used usernames that do not exist on the server: ```bash sudo grep 'Invalid user' /var/log/auth.log | awk '{for (i=1; i<=NF; i++) if ($i == "user") print $(i+1)}' | sort | uniq -c | sort -rn | head ``` List successful logins, which is what matters most in an investigation: ```bash sudo grep 'Accepted' /var/log/auth.log ``` ```output 2026-09-25T08:03:11.482915+00:00 web01 sshd[1834]: Accepted publickey for your_user from 192.0.2.10 port 51844 ssh2: ED25519 SHA256:... ``` Every `Accepted` line should come from a user and IP address you recognize. Also check who ran commands with `sudo`: ```bash sudo grep 'sudo:' /var/log/auth.log | grep 'COMMAND=' ``` The `last` command summarizes login sessions, including reboots: ```bash last -n 10 ``` If you see thousands of failed attempts, disable password authentication in SSH and use key-based logins only, or install Fail2ban to block offending IPs automatically. ## Step 5 - Analyzing web server access logs Application logs follow their own formats. Nginx and Apache use the combined log format by default, where each request is one line and the fields are separated by spaces. That makes `awk` a quick analysis tool. The examples use `/var/log/nginx/access.log`; for Apache, use `/var/log/apache2/access.log`. The 10 IP addresses making the most requests: ```bash sudo awk '{print $1}' /var/log/nginx/access.log | sort | uniq -c | sort -rn | head ``` Requests per HTTP status code. A jump in `5xx` codes means the application is failing; many `404` codes often mean a scanner is probing for known paths: ```bash sudo awk '{print $9}' /var/log/nginx/access.log | sort | uniq -c | sort -rn ``` ```output 18542 200 1204 304 873 404 61 502 ``` The URLs that returned server errors: ```bash sudo awk '$9 ~ /^5/ {print $7}' /var/log/nginx/access.log | sort | uniq -c | sort -rn | head ``` When you find `502` errors, look at the Nginx error log and at the backend service it proxies to at the same timestamps: ```bash sudo tail -n 50 /var/log/nginx/error.log ``` ## Step 6 - Managing log rotation and disk usage Logs grow continuously. Two mechanisms keep them in check on Ubuntu: logrotate for text files and journald's own size limits for the journal. Check how much space logs use: ```bash sudo du -sh /var/log sudo journalctl --disk-usage ``` ```output 412M /var/log Archived and active journals take 296.0M in the file system. ``` logrotate runs daily from a systemd timer and reads one configuration file per package in `/etc/logrotate.d/`. Look at the one for rsyslog files: ```bash cat /etc/logrotate.d/rsyslog ``` To add rotation for your own application's logs in `/var/log/myapp/`, create a file: ```bash sudo nano /etc/logrotate.d/myapp ``` ```text /var/log/myapp/*.log { daily rotate 14 compress delaycompress missingok notifempty copytruncate } ``` This keeps 14 daily copies, compresses all but the most recent and skips empty or missing files. `copytruncate` lets the application keep writing to the same file descriptor; if your application can reopen its log on a signal, use a `postrotate` script instead. Test the configuration in debug mode, which shows what would happen without changing anything: ```bash sudo logrotate -d /etc/logrotate.d/myapp ``` For the journal, set a maximum size so it cannot fill the disk. Create a drop-in file: ```bash sudo mkdir -p /etc/systemd/journald.conf.d sudo nano /etc/systemd/journald.conf.d/size.conf ``` ```ini [Journal] SystemMaxUse=500M ``` Apply it and verify: ```bash sudo systemctl restart systemd-journald sudo journalctl --disk-usage ``` To free space immediately, remove archived journal files older than two weeks: ```bash sudo journalctl --vacuum-time=2weeks ``` > Warning: Never delete an open log file with `rm` to free space. The process keeps writing to the deleted file and the space is not released until it restarts. Use logrotate, or empty the file with `sudo truncate -s 0 /path/to/file.log`. ## Troubleshooting - **`journalctl -b -1` says `Specified boot ID not found`**: the journal is not persistent or the previous boot has been rotated out. Check that `/var/log/journal` exists and that `SystemMaxUse` is not too small. - **`/var/log/syslog` or `/var/log/auth.log` is missing**: rsyslog is not installed or not running. Check with `systemctl status rsyslog`, or use `journalctl` instead. - **`Permission denied` when reading a log**: the file is owned by `root` or `syslog` with group `adm`. Use `sudo` or add your user to `adm` as shown in Step 1. - **A log file keeps growing after rotation**: the application still writes to the old, renamed file. Add `copytruncate` or a `postrotate` command that tells the service to reopen its logs. ## Conclusion You now know where Ubuntu 24.04 keeps its logs, how to search text logs with `tail`, `grep` and `zgrep`, how to filter the systemd journal by service, time, boot and priority, and how to use `awk` to turn raw log lines into answers about SSH attacks and web errors. As next steps, harden SSH and install Fail2ban based on what `auth.log` shows, and ship logs to a central system such as Loki or Graylog so you can search across servers and keep history after a disk failure.