--- title: "How to Configure journald and Analyze Logs with journalctl" description: "Make the systemd journal persistent, cap its disk usage and learn journalctl filters by unit, time, boot, priority and field to troubleshoot Ubuntu 24.04." category: "Monitoring Logging" language: en source: https://cubepath.com/docs/monitoring-logging/journalctl-configuration-for-log-analysis updated: 2026-09-25 --- # How to Configure journald and Analyze Logs with journalctl systemd-journald collects logs from the kernel, from every systemd service and from syslog into a single indexed journal, and `journalctl` is the tool to query it. Because each entry carries structured fields such as the unit, PID and priority, you can filter much more precisely than with plain text files. In this tutorial you will configure journald on Ubuntu 24.04 so logs survive reboots without filling the disk, then use `journalctl` to filter by service, time, boot, priority and field, and finally export and clean up the journal. ## Prerequisites To follow this tutorial you need: - A server running Ubuntu 24.04 LTS, such as a CubePath VPS. The commands also work on Debian 12 and Rocky Linux 9; the differences are noted where they matter. - A non-root user with `sudo` privileges. Regular users only see their own logs. To read the whole system journal without `sudo`, add your user to the `systemd-journal` group (members of `adm` can also read it on Ubuntu) and log in again: ```bash sudo usermod -aG systemd-journal your_user ``` The examples below still use `sudo` so they work either way. ## Step 1 - Enabling persistent storage journald keeps logs either in memory under `/run/log/journal`, which is lost on reboot, or on disk under `/var/log/journal`. The default setting, `Storage=auto`, stores logs on disk only if `/var/log/journal` exists. Ubuntu creates that directory, while Debian 12 and some minimal images do not. Check whether you already have logs from previous boots: ```bash sudo journalctl --list-boots ``` ```output IDX BOOT ID FIRST ENTRY LAST ENTRY -2 5f1c0e5a9c2d4b1d8f6a3e7b9c1d2e3f Mon 2026-09-15 08:11:02 UTC Tue 2026-09-23 17:45:10 UTC -1 0a9b8c7d6e5f4a3b2c1d0e9f8a7b6c5d Tue 2026-09-23 17:46:01 UTC Thu 2026-09-25 09:58:40 UTC 0 3e2d1c0b9a8f7e6d5c4b3a2f1e0d9c8b Thu 2026-09-25 10:00:12 UTC Thu 2026-09-25 12:31:55 UTC ``` If only boot `0` is listed on a server that has been rebooted before, the journal is volatile. Rather than relying on the directory existing, make persistence explicit with a drop-in file. Drop-ins in `/etc/systemd/journald.conf.d/` override the defaults without editing the packaged `journald.conf`: ```bash sudo mkdir -p /etc/systemd/journald.conf.d sudo nano /etc/systemd/journald.conf.d/10-persistent.conf ``` ```ini [Journal] Storage=persistent ``` Restart journald so it creates `/var/log/journal` and starts writing there: ```bash sudo systemctl restart systemd-journald ls /var/log/journal/ ``` ```output d3f1a2b4c5d6e7f8091a2b3c4d5e6f70 ``` The directory name is the machine ID. From now on, `journalctl --list-boots` will show one line per boot. ## Step 2 - Limiting the journal size and retention With persistent storage, journald by default uses up to 10% of the file system (capped at 4 GB) and always leaves 15% free. On a small VPS disk you may want a fixed limit and a maximum age. Create a second drop-in: ```bash sudo nano /etc/systemd/journald.conf.d/20-retention.conf ``` ```ini [Journal] SystemMaxUse=1G SystemKeepFree=2G SystemMaxFileSize=128M MaxRetentionSec=1month ``` - `SystemMaxUse=1G`: the journal on disk never grows beyond 1 GB. - `SystemKeepFree=2G`: journald stops using space when less than 2 GB would remain free on the file system. - `SystemMaxFileSize=128M`: individual journal files are rotated at this size, so old data can be removed in reasonable chunks. - `MaxRetentionSec=1month`: entries older than one month are deleted even if there is space left. Apply the change and check the effective configuration, which merges `journald.conf` with all drop-ins: ```bash sudo systemctl restart systemd-journald systemd-analyze cat-config systemd/journald.conf | grep -v '^#' | grep . ``` ```output [Journal] [Journal] Storage=persistent [Journal] SystemMaxUse=1G SystemKeepFree=2G SystemMaxFileSize=128M MaxRetentionSec=1month ``` Check how much space the journal uses right now: ```bash sudo journalctl --disk-usage ``` ```output Archived and active journals take 312.0M in the file system. ``` ## Step 3 - Viewing and following logs Running `journalctl` without options shows the entire journal, oldest first, in a pager. In practice you almost always add a few options: ```bash # Last 50 entries, no pager sudo journalctl -n 50 --no-pager # Follow new entries in real time sudo journalctl -f # Jump to the end of the journal inside the pager sudo journalctl -e # Newest entries first sudo journalctl -r ``` Add `-x` to show explanatory help text from the message catalog next to some entries, which is useful when a unit fails. ## Step 4 - Filtering by unit, time and boot The most common filter is the systemd unit. On Ubuntu 24.04 the SSH server unit is called `ssh`; on Rocky Linux it is `sshd`: ```bash sudo journalctl -u ssh sudo journalctl -u nginx -u php8.3-fpm ``` When you pass several `-u` options, entries from any of them are shown, interleaved by time. Filter by time with `--since` and `--until`. Both accept absolute timestamps and relative expressions: ```bash sudo journalctl --since "2026-09-25 09:00" --until "2026-09-25 10:30" sudo journalctl --since "1 hour ago" sudo journalctl --since yesterday --until today sudo journalctl -u nginx --since "15 min ago" ``` Filter by boot with `-b`. Without an argument it means the current boot; `-1` is the previous one. This is how you find out why a server rebooted or crashed: ```bash sudo journalctl -b -1 -e ``` The last lines of the previous boot show whether the shutdown was clean (services stopping one by one) or abrupt (the log simply ends). Kernel messages, the same ones `dmesg` shows, are selected with `-k`: ```bash sudo journalctl -k -b ``` ## Step 5 - Filtering by priority and message content Every entry has a syslog priority from 0 (`emerg`) to 7 (`debug`). `-p` shows entries with the given priority or more severe: | Priority | Name | Typical use | |---|---|---| | 0 | `emerg` | System is unusable | | 1 | `alert` | Immediate action required | | 2 | `crit` | Critical conditions | | 3 | `err` | Errors | | 4 | `warning` | Warnings | | 5 | `notice` | Normal but significant | | 6 | `info` | Informational | | 7 | `debug` | Debug messages | ```bash # Errors and worse in the current boot sudo journalctl -p err -b # Only warnings, not errors sudo journalctl -p warning..warning --since today ``` To search inside messages, use `-g` (`--grep`). It takes a regular expression and is case-insensitive when the pattern is all lowercase: ```bash sudo journalctl -u ssh -g 'failed password' --since today sudo journalctl -k -g 'out of memory|oom-kill' ``` ```output Sep 25 11:42:07 web01 kernel: Out of memory: Killed process 20431 (php-fpm8.3) total-vm:812344kB, anon-rss:402112kB ``` ## Step 6 - Filtering by journal fields Each journal entry is a set of fields. Fields starting with `_` are added by journald itself and cannot be faked by the application. Display every field of the latest entry of a unit with the `verbose` output: ```bash sudo journalctl -u nginx -n 1 -o verbose ``` Use `FIELD=value` arguments to filter on any of them: ```bash # Everything logged by one process sudo journalctl _PID=1422 # Everything logged by a program, whatever unit started it sudo journalctl _COMM=sshd # Messages with a given syslog identifier (same as -t sshd) sudo journalctl SYSLOG_IDENTIFIER=sshd # Everything logged by processes running as UID 1000 sudo journalctl _UID=1000 ``` Different fields are combined with AND, and repeating the same field combines the values with OR. To discover which values exist for a field, use `-F`: ```bash sudo journalctl -F _SYSTEMD_UNIT | sort | head ``` ```output cron.service nginx.service ssh.service systemd-journald.service systemd-logind.service ``` ## Step 7 - Changing the output format and exporting logs `-o` changes the output format. The most useful ones are: | Format | Use | |---|---| | `short-iso` | Like the default, with ISO 8601 timestamps | | `cat` | Only the message text, good for piping to `grep` or `awk` | | `json` | One JSON object per line, for scripts and log shippers | | `json-pretty` | Indented JSON, for reading | | `export` | Binary-safe format to move journal data to another machine | Export one service's logs from yesterday as JSON to attach to a ticket or feed to another tool: ```bash sudo journalctl -u nginx --since yesterday --until today -o json --no-pager > nginx-yesterday.json ``` Because each line is a JSON object, `jq` can process the file. Install it with `sudo apt install jq`, then, for example, count messages per priority: ```bash jq -r '.PRIORITY' nginx-yesterday.json | sort | uniq -c ``` To keep a full copy of the journal for later analysis, including all fields, copy the journal files themselves and open them with `--directory` on any machine with systemd: ```bash sudo tar -czf journal-web01.tar.gz -C /var/log/journal . ``` ```bash mkdir journal-web01 && tar -xzf journal-web01.tar.gz -C journal-web01 journalctl --directory=journal-web01 -p err ``` ## Step 8 - Cleaning up the journal manually The limits from Step 2 are enforced automatically, but sometimes you need to free space right away, for example when a disk is almost full. `--vacuum-*` deletes archived journal files until the condition is met: ```bash # Keep only the last two weeks sudo journalctl --vacuum-time=2weeks # Shrink the journal to 500 MB sudo journalctl --vacuum-size=500M ``` ```output Deleted archived journal /var/log/journal/d3f1a2b4.../system@0006...-0006....journal~ (128.0M). Vacuuming done, freed 384.0M of archived journals from /var/log/journal/d3f1a2b4.... ``` Vacuuming only removes archived files, not the one being written. To include recent data, rotate first so the active file is archived: ```bash sudo journalctl --rotate sudo journalctl --vacuum-time=1d ``` Check the journal files for corruption with: ```bash sudo journalctl --verify ``` Each file is listed as `PASS`. A file reported as `FAIL`, usually after a hard power loss, can be rotated away; journald keeps writing to a new file. ## Troubleshooting - **`--list-boots` shows only the current boot.** Storage is still volatile. Check that `Storage=persistent` appears in `systemd-analyze cat-config systemd/journald.conf` and that journald was restarted. - **The journal uses more space than expected.** Other limits are also in play: whichever of `SystemMaxUse` and `SystemKeepFree` is stricter wins. Confirm with `sudo journalctl --disk-usage` after a `sudo journalctl --rotate`. - **Entries are missing and the journal shows `Suppressed N messages`.** journald rate-limits each service, by default 10,000 messages per 30 seconds. Reduce the verbosity of the application, or raise the limit for everything with `RateLimitBurst=` and `RateLimitIntervalSec=` in a journald drop-in, or for one service with `LogRateLimitBurst=` and `LogRateLimitIntervalSec=` in its unit override. - **Without `sudo` you only see your own messages** and journalctl prints a hint that you are not seeing messages from other users and the system. Your user is not in the `systemd-journal` or `adm` group, or you have not logged in again after being added. ## Conclusion Your journal is now persistent, limited in size and age, and you can find any log line by unit, time, boot, priority, message text or journal field, export it as JSON and clean up space on demand. As next steps, use `journalctl -o cat` together with `grep` and `awk` for deeper text analysis, forward logs from several servers to a central rsyslog server, or combine `journalctl -u` with `systemctl status` to monitor and troubleshoot individual services.