systemd is the init system and service manager on Ubuntu, Debian, Rocky Linux and most other distributions. It describes everything it manages (services, sockets, timers, mount points, targets) as declarative unit files and starts them according to their dependencies. In this tutorial you will build a small hardened service on Ubuntu 24.04 (systemd 255) and use it to learn how unit files, dependencies, targets, drop-in overrides, socket activation, resource limits, timers and the journal work.
Prerequisites
To follow this tutorial you need:
- A server running Ubuntu 24.04 LTS, for example a CubePath VPS. The commands also work on Debian 12 and Rocky Linux 9.
- A non-root user with
sudoprivileges. - Python 3, which Ubuntu 24.04 includes by default, for the example service.
Unit types and where they live
Every unit has a type given by its file extension:
| Type | Extension | Manages |
|---|---|---|
| Service | .service | Daemons and one-shot commands |
| Socket | .socket | Listening sockets that start a service on demand |
| Target | .target | Groups of units, used as synchronization points |
| Timer | .timer | Scheduled activation of another unit |
| Mount | .mount | Filesystem mount points |
| Path | .path | Activation when a file or directory changes |
| Slice | .slice | cgroup groups for resource control |
systemd reads unit files from several directories. When the same name exists in more than one, the first in this list wins:
| Directory | Owner |
|---|---|
/etc/systemd/system/ | The administrator (your custom units and overrides) |
/run/systemd/system/ | Runtime units, lost on reboot |
/usr/lib/systemd/system/ | Units installed by packages; do not edit |
Step 1 - Inspecting existing units
Before writing your own unit, look at how installed ones are defined. List enabled unit files:
systemctl list-unit-files --type=service --state=enabled
Show the file for a unit, including any drop-in overrides, with systemctl cat:
systemctl cat ssh.service
# /usr/lib/systemd/system/ssh.service
[Unit]
Description=OpenBSD Secure Shell server
Documentation=man:sshd(8) man:sshd_config(5)
After=network.target nss-user-lookup.target auditd.service
...
To see the effective value of any property, including defaults that are not in the file, use systemctl show:
systemctl show ssh.service -p Restart -p MainPID -p ActiveState
Restart=on-failure
MainPID=880
ActiveState=active
Step 2 - Writing a hardened service unit
Create a small web application to manage. It serves a directory over HTTP on port 8080 using Python's built-in server. First create a dedicated system user and the content directory:
sudo useradd --system --no-create-home --shell /usr/sbin/nologin demoapp
sudo mkdir -p /srv/demoapp
echo "hello from systemd" | sudo tee /srv/demoapp/index.html
Create the unit file:
sudo nano /etc/systemd/system/demoapp.service
[Unit]
Description=Demo HTTP application
Documentation=https://docs.python.org/3/library/http.server.html
Wants=network-online.target
After=network-online.target
[Service]
Type=exec
User=demoapp
Group=demoapp
WorkingDirectory=/srv/demoapp
Environment=PYTHONUNBUFFERED=1
ExecStart=/usr/bin/python3 -m http.server 8080 --bind 127.0.0.1 --directory /srv/demoapp
Restart=on-failure
RestartSec=5s
# Hardening
NoNewPrivileges=yes
ProtectSystem=strict
ProtectHome=yes
PrivateTmp=yes
PrivateDevices=yes
ProtectKernelTunables=yes
ProtectKernelModules=yes
ProtectControlGroups=yes
RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX
[Install]
WantedBy=multi-user.target
The sections have distinct roles:
[Unit]holds metadata and dependencies.Wants=network-online.targetplusAfter=waits until the network is configured, not just started.[Service]defines how to run the process.Type=exectells systemd the service is up once the binary has been executed successfully, andRestart=on-failurerestarts it after a crash but not after a clean stop.- The hardening options make the whole filesystem read-only for the service (
ProtectSystem=strict), hide/home, and give it a private/tmp. [Install]is only used bysystemctl enable: it creates a link somulti-user.targetpulls in the service at boot.
Reload systemd so it reads the new file, then enable and start the service:
sudo systemctl daemon-reload
sudo systemctl enable --now demoapp.service
Verify that it runs and answers:
systemctl status demoapp.service --no-pager
curl -s http://127.0.0.1:8080/
● demoapp.service - Demo HTTP application
Loaded: loaded (/etc/systemd/system/demoapp.service; enabled; preset: enabled)
Active: active (running) since Fri 2026-09-25 10:20:11 UTC; 3s ago
Main PID: 2210 (python3)
hello from systemd
systemd can score how exposed a service is. Compare your unit with an unhardened one:
systemd-analyze security demoapp.service | tail -n 1
→ Overall exposure level for demoapp.service: 4.6 OK
Lower is better; a service with no hardening scores around 9.6 (UNSAFE). Your exact number depends on the options you set.
Step 3 - Understanding dependencies and ordering
systemd separates two questions: which units must be started together (requirement dependencies), and in which order (ordering dependencies). Setting one does not imply the other.
| Directive | Effect |
|---|---|
Wants= | Start the listed unit too; keep going if it fails. The recommended default |
Requires= | Start the listed unit too; if it fails to start, or is stopped explicitly, this unit is stopped |
BindsTo= | Like Requires=, and also stop this unit if the other one stops for any reason |
PartOf= | Stop or restart this unit when the listed unit is stopped or restarted |
After= / Before= | Ordering only: wait for the listed unit to finish starting |
Conflicts= | Starting this unit stops the listed one, and the reverse |
A common mistake is Requires=mysql.service without After=mysql.service: both units start in parallel and the application may connect before the database is ready. Always pair a requirement with an ordering directive.
See what a unit pulls in:
systemctl list-dependencies demoapp.service
And which units depend on a given one, for example which services need the network online:
systemctl list-dependencies --reverse network-online.target
To analyze boot time, critical-chain shows the chain of units that delayed a target, and blame lists units by start time:
systemd-analyze critical-chain demoapp.service
systemd-analyze blame | head -n 5
demoapp.service +45ms
└─network-online.target @3.912s
└─systemd-networkd-wait-online.service @1.203s +2.707s
└─systemd-networkd.service @1.050s +140ms
The @ value is when the unit became active and + is how long it took to start.
Step 4 - Changing units safely with drop-in overrides
Never edit files in /usr/lib/systemd/system/: package upgrades overwrite them. Use a drop-in file instead, which systemd merges on top of the original. systemctl edit creates it for you:
sudo systemctl edit demoapp.service
The editor opens a file at /etc/systemd/system/demoapp.service.d/override.conf. Add only the settings you want to change between the comment markers:
[Service]
Environment=APP_ENV=production
RestartSec=10s
Save and exit; systemctl edit reloads the configuration automatically. Restart the service and check the merged result:
sudo systemctl restart demoapp.service
systemctl cat demoapp.service | tail -n 4
systemctl show demoapp.service -p RestartUSec -p Environment
RestartUSec=10s
Environment=PYTHONUNBUFFERED=1 APP_ENV=production
List-type settings such as ExecStart= are appended rather than replaced. To replace one, clear it first with an empty assignment:
[Service]
ExecStart=
ExecStart=/usr/bin/python3 -m http.server 9090 --bind 127.0.0.1 --directory /srv/demoapp
To remove all overrides, run sudo systemctl revert demoapp.service.
Step 5 - Working with targets
Targets group units and replace SysV runlevels. multi-user.target is a normal server without a graphical session, graphical.target adds a display manager, and rescue.target is single-user maintenance mode. Check the default:
systemctl get-default
multi-user.target
Servers should boot into multi-user.target. If a server was installed with a desktop and you want to stop starting it:
sudo systemctl set-default multi-user.target
systemctl isolate switches the running system to another target and stops everything that target does not include.
Warning
sudo systemctl isolate rescue.targetstops networking and SSH. Only use it from a local or out-of-band console.
List active targets to see what is reached at the moment:
systemctl list-units --type=target
Step 6 - Starting services on demand with socket activation
With socket activation, systemd opens the listening socket itself and starts the service only when the first connection arrives. It also lets you restart the service without dropping connections waiting in the queue. The simplest example is an echo service where systemd starts one process per connection (Accept=yes) and connects the socket to its standard input and output.
Create the socket unit:
sudo nano /etc/systemd/system/echo.socket
[Unit]
Description=Echo service socket
[Socket]
ListenStream=127.0.0.1:7777
Accept=yes
[Install]
WantedBy=sockets.target
With Accept=yes the service must be a template (note the @), because systemd creates one instance per connection:
sudo nano /etc/systemd/system/[email protected]
[Unit]
Description=Echo service connection
[Service]
ExecStart=/usr/bin/cat
StandardInput=socket
DynamicUser=yes
Enable only the socket; the service starts by itself:
sudo systemctl daemon-reload
sudo systemctl enable --now echo.socket
systemctl list-sockets echo.socket
The output lists 127.0.0.1:7777 with echo.socket as its unit. No echo@ service is running yet.
Connect and type a line; it is echoed back. Press Ctrl+C to close:
nc 127.0.0.1 7777
For long-running daemons, use Accept=no (the default). systemd then passes the listening socket to a single service instance, which must support receiving it (for example through sd_listen_fds()).
Step 7 - Limiting resources
Every service runs in its own cgroup, so you can limit it directly in the unit. Add limits to the demo service with a drop-in:
sudo systemctl edit demoapp.service
[Service]
CPUQuota=50%
MemoryHigh=200M
MemoryMax=256M
TasksMax=64
LimitNOFILE=4096
CPUQuota=50% allows half of one CPU. MemoryHigh throttles and reclaims memory above 200 MB, and MemoryMax is the hard limit that triggers the OOM killer. TasksMax caps processes plus threads. Restart and verify:
sudo systemctl restart demoapp.service
systemctl show demoapp.service -p CPUQuotaPerSecUSec -p MemoryMax -p TasksMax
CPUQuotaPerSecUSec=500ms
MemoryMax=268435456
TasksMax=64
To change a limit on a running service without restarting it, use set-property. The --runtime flag makes the change temporary:
sudo systemctl set-property --runtime demoapp.service MemoryMax=300M
Watch live usage per cgroup with systemd-cgtop (press q to quit).
Step 8 - Replacing cron with a timer
Timers run a service on a schedule, log to the journal and can catch up on runs missed while the server was off. Create a service that removes files older than seven days from a temporary directory:
sudo mkdir -p /srv/demoapp-tmp
sudo nano /etc/systemd/system/cleanup-demoapp.service
[Unit]
Description=Remove old files from /srv/demoapp-tmp
[Service]
Type=oneshot
ExecStart=/usr/bin/find /srv/demoapp-tmp -type f -mtime +7 -delete
Create the timer with the same name:
sudo nano /etc/systemd/system/cleanup-demoapp.timer
[Unit]
Description=Daily cleanup of /srv/demoapp-tmp
[Timer]
OnCalendar=*-*-* 03:00:00
RandomizedDelaySec=10m
Persistent=true
[Install]
WantedBy=timers.target
Persistent=true runs the job at boot if the 03:00 run was missed. Check the calendar expression before enabling the timer:
systemd-analyze calendar "*-*-* 03:00:00"
Original form: *-*-* 03:00:00
Normalized form: *-*-* 03:00:00
Next elapse: Sat 2026-09-26 03:00:00 UTC
From now: 16h left
Enable the timer (not the service) and list it:
sudo systemctl daemon-reload
sudo systemctl enable --now cleanup-demoapp.timer
systemctl list-timers cleanup-demoapp.timer
Run the job once by hand to test it: sudo systemctl start cleanup-demoapp.service.
Step 9 - Reading logs with journalctl
Everything a service writes to stdout and stderr ends up in the journal. Ubuntu 24.04 stores it persistently in /var/log/journal. The most useful filters:
journalctl -u demoapp.service -n 50 --no-pager
Follow new entries in real time:
journalctl -u demoapp.service -f
Only messages from the current boot, or from the previous one (useful after a crash):
journalctl -b -u demoapp.service
journalctl -b -1 -p err
-p err shows priority err and higher (crit, alert, emerg). Filter by time and export JSON for processing:
journalctl -u demoapp.service --since "1 hour ago" -o json-pretty | head -n 20
Check how much disk the journal uses and trim it if needed:
journalctl --disk-usage
sudo journalctl --vacuum-time=30d
Troubleshooting
status=203/EXEC. systemd could not execute ExecStart. The path must be absolute and the file executable; check with ls -l /path/to/binary. With ProtectSystem=strict or ProtectHome=yes, the binary may be in a hidden or read-only location.
status=217/USER. The user in User= does not exist. Create it or fix the name.
Changes to a unit file are ignored. Run sudo systemctl daemon-reload after editing any unit file by hand. systemctl status warns when the file on disk changed and needs a reload.
The service starts before the network is ready. Use both Wants=network-online.target and After=network-online.target. network.target only means the network management stack has started, not that interfaces have addresses.
A hardened service cannot write its data. ProtectSystem=strict makes everything read-only. Grant specific paths with ReadWritePaths=/var/lib/demoapp, or let systemd create one with StateDirectory=demoapp.
Conclusion
You wrote and hardened a service unit, learned how requirement and ordering dependencies differ, changed units safely with drop-ins, switched targets, used socket activation, applied cgroup resource limits, replaced a cron job with a timer and filtered the journal. Next, explore systemd-analyze security recommendations for your real services, read man systemd.exec and man systemd.resource-control for every available option, and move your remaining cron jobs to timers.
