rsyslog is the syslog daemon shipped with Ubuntu and Debian. Besides writing local log files, it can receive messages from other machines and forward its own messages to a remote server, which makes it a simple way to keep the logs of several servers in one place. In this tutorial you will turn one Ubuntu 24.04 server into a central log server, configure a client to forward all of its logs over TCP with a disk-backed queue, encrypt the connection with TLS and set up rotation for the collected files.
Prerequisites
To follow this tutorial you need:
- Two servers running Ubuntu 24.04 LTS, for example two CubePath VPS: one will be the log server and the other the client. You can add more clients later by repeating the client steps.
- A non-root user with
sudoprivileges on both servers. - Network connectivity between them, ideally over a private network. In the examples the log server has the IP
your_log_server_ipand the clients live in the subnetyour_client_subnet(for example10.0.0.0/24). - Enough free disk space on the log server for the volume of logs you expect. A few GB is plenty for a handful of servers with default logging.
- For the TLS step: a DNS name that resolves to the log server, called
logs.example.comin this guide. An entry in/etc/hostson each client also works.
Step 1 - Checking that rsyslog is installed
Ubuntu 24.04 server images include rsyslog, but minimal or container-derived images may not. Run the following on both servers:
rsyslogd -v | head -n 1
systemctl is-active rsyslog
rsyslogd 8.2312.0 (aka 2023.12) compiled with:
active
If the command is not found, install and start it:
sudo apt update
sudo apt install rsyslog
sudo systemctl enable --now rsyslog
On Ubuntu, rsyslog starts as root and then drops privileges to the syslog user. Any directory it writes to must therefore be writable by syslog, which matters in the next step.
Step 2 - Configuring the central log server
On the log server, create the directory that will hold the remote logs, owned by syslog:
sudo install -d -o syslog -g adm -m 0750 /var/log/remote
Next, create a configuration file that loads the TCP input module and stores every remote message in /var/log/remote/<hostname>/<program>.log. Using a dedicated ruleset keeps remote messages out of the server's own /var/log/syslog:
sudo nano /etc/rsyslog.d/10-remote-server.conf
# Receive syslog over TCP
module(load="imtcp")
# One directory per sending host, one file per program
template(name="RemoteLogs" type="string"
string="/var/log/remote/%HOSTNAME:::secpath-replace%/%PROGRAMNAME:::secpath-replace%.log")
ruleset(name="remote") {
action(type="omfile"
dynaFile="RemoteLogs"
dirCreateMode="0750"
fileCreateMode="0640")
}
input(type="imtcp" port="514" ruleset="remote")
The secpath-replace option replaces slashes and other unsafe characters in the hostname and program name, so a client cannot write outside /var/log/remote by sending a crafted hostname.
TCP is used instead of UDP because it tells the client when the server is unreachable, which lets the client queue messages instead of losing them.
Check the syntax before restarting:
sudo rsyslogd -N1
rsyslogd: version 8.2312.0, config validation run (level 1), master config /etc/rsyslog.conf
rsyslogd: End of config validation run. Bye.
Restart rsyslog and confirm it listens on port 514:
sudo systemctl restart rsyslog
sudo ss -tlnp | grep ':514'
LISTEN 0 25 0.0.0.0:514 0.0.0.0:* users:(("rsyslogd",pid=2143,fd=6))
LISTEN 0 25 [::]:514 [::]:* users:(("rsyslogd",pid=2143,fd=7))
Syslog over plain TCP has no authentication, so only allow your client subnet through the firewall. If UFW is enabled on the log server:
sudo ufw allow from your_client_subnet to any port 514 proto tcp
sudo ufw status
WarningDo not open port 514 to the whole Internet. Anyone who can reach it can write arbitrary messages into your logs and fill the disk. If clients connect over the public Internet, restrict the source IPs and use TLS as shown in Step 5.
Step 3 - Forwarding logs from a client
On the client, create a forwarding rule. The omfwd action sends every message to the log server, and the queue settings make the forwarding reliable: if the server is down, messages are buffered in memory and spilled to disk under /var/spool/rsyslog (the work directory already set in Ubuntu's /etc/rsyslog.conf) until the connection comes back.
sudo nano /etc/rsyslog.d/90-forward.conf
# Forward everything to the central log server over TCP
action(type="omfwd"
target="your_log_server_ip"
port="514"
protocol="tcp"
queue.type="LinkedList"
queue.filename="fwd_central"
queue.maxDiskSpace="1g"
queue.saveOnShutdown="on"
action.resumeRetryCount="-1")
The options mean:
queue.type="LinkedList"withqueue.filenamecreates a disk-assisted queue, so messages survive short outages.queue.maxDiskSpace="1g"caps the disk used by the queue.queue.saveOnShutdown="on"writes pending messages to disk when rsyslog stops.action.resumeRetryCount="-1"retries forever instead of discarding messages.
Local logging is not affected: the default rules in /etc/rsyslog.d/50-default.conf still write /var/log/syslog and /var/log/auth.log on the client.
Validate and restart:
sudo rsyslogd -N1
sudo systemctl restart rsyslog
To forward only part of the logs, put a filter in front of the action. For example, to send only authentication messages and anything with severity warning or higher, wrap the action like this:
if ($syslogfacility-text == "auth" or $syslogfacility-text == "authpriv" or $syslogseverity <= 4) then {
action(type="omfwd" target="your_log_server_ip" port="514" protocol="tcp"
queue.type="LinkedList" queue.filename="fwd_central"
queue.saveOnShutdown="on" action.resumeRetryCount="-1")
}
Step 4 - Testing log forwarding
On the client, write a test message with logger:
logger -t rsyslog-test "Hello from $(hostname)"
On the log server, a directory named after the client should appear with a file for the rsyslog-test tag:
sudo ls /var/log/remote/
sudo tail -n 3 /var/log/remote/client01/rsyslog-test.log
client01
2026-09-25T10:14:02.118402+00:00 client01 rsyslog-test: Hello from client01
Replace client01 with the hostname of your client. Other files, such as sshd.log, sudo.log or CRON.log, will appear as those programs log activity.
To watch everything arriving from all clients in real time:
sudo tail -F /var/log/remote/*/*.log
You can also test the queue: stop rsyslog on the log server, send a few messages with logger on the client, then start rsyslog again. The messages arrive a few seconds after the server is back.
Step 5 - Encrypting the connection with TLS
Plain TCP sends log lines in clear text. If logs travel over a network you do not fully control, encrypt them with TLS. rsyslog uses a separate package for the GnuTLS driver; install it on both servers:
sudo apt install rsyslog-gnutls
Creating a CA and a server certificate
This example uses a small private certificate authority (CA) on the log server. Clients will trust this CA and check that the server certificate matches logs.example.com.
On the log server, create a directory readable by the syslog group. Because your user cannot enter it, the following commands use absolute paths with sudo:
sudo install -d -o root -g syslog -m 0750 /etc/rsyslog.d/tls
Generate the CA key and certificate:
sudo openssl req -x509 -newkey rsa:3072 -nodes -days 3650 \
-keyout /etc/rsyslog.d/tls/ca-key.pem -out /etc/rsyslog.d/tls/ca.pem \
-subj "/CN=rsyslog-ca"
Generate the server key and a certificate signing request:
sudo openssl req -newkey rsa:3072 -nodes \
-keyout /etc/rsyslog.d/tls/server-key.pem -out /etc/rsyslog.d/tls/server.csr \
-subj "/CN=logs.example.com"
Sign it with the CA, adding the DNS name as a subject alternative name, which is what clients check:
echo "subjectAltName=DNS:logs.example.com" | sudo tee /etc/rsyslog.d/tls/server.ext
sudo openssl x509 -req -in /etc/rsyslog.d/tls/server.csr \
-CA /etc/rsyslog.d/tls/ca.pem -CAkey /etc/rsyslog.d/tls/ca-key.pem -CAcreateserial \
-days 825 -extfile /etc/rsyslog.d/tls/server.ext -out /etc/rsyslog.d/tls/server-cert.pem
Certificate request self-signature ok
subject=CN = logs.example.com
Set ownership and permissions. rsyslog needs to read server-key.pem, while the CA key must stay readable by root only:
sudo chown root:syslog /etc/rsyslog.d/tls/server-key.pem /etc/rsyslog.d/tls/server-cert.pem /etc/rsyslog.d/tls/ca.pem
sudo chmod 0640 /etc/rsyslog.d/tls/server-key.pem
sudo chmod 0600 /etc/rsyslog.d/tls/ca-key.pem
TipKeep the certificates under
/etc/rsyslog.d/. The rsyslog AppArmor profile in Ubuntu allows reading that directory, while other locations such as/etc/ssl/privatemay be denied.
Switching the server to TLS
Replace the content of /etc/rsyslog.d/10-remote-server.conf so the TCP input uses TLS on the standard port 6514:
sudo nano /etc/rsyslog.d/10-remote-server.conf
global(
DefaultNetstreamDriverCAFile="/etc/rsyslog.d/tls/ca.pem"
DefaultNetstreamDriverCertFile="/etc/rsyslog.d/tls/server-cert.pem"
DefaultNetstreamDriverKeyFile="/etc/rsyslog.d/tls/server-key.pem"
)
# TLS-only TCP input, clients are not required to present a certificate
module(load="imtcp"
StreamDriver.Name="gtls"
StreamDriver.Mode="1"
StreamDriver.AuthMode="anon")
template(name="RemoteLogs" type="string"
string="/var/log/remote/%HOSTNAME:::secpath-replace%/%PROGRAMNAME:::secpath-replace%.log")
ruleset(name="remote") {
action(type="omfile"
dynaFile="RemoteLogs"
dirCreateMode="0750"
fileCreateMode="0640")
}
input(type="imtcp" port="6514" ruleset="remote")
StreamDriver.Mode="1" makes every connection on this input TLS-only. AuthMode="anon" means the server does not verify client certificates; the clients still verify the server. Restricting which IPs can connect is done with the firewall.
Validate, restart and open the new port, then remove the plain-text rule:
sudo rsyslogd -N1
sudo systemctl restart rsyslog
sudo ufw allow from your_client_subnet to any port 6514 proto tcp
sudo ufw delete allow from your_client_subnet to any port 514 proto tcp
Check the certificate from the server itself:
sudo openssl s_client -connect localhost:6514 -CAfile /etc/rsyslog.d/tls/ca.pem \
-verify_hostname logs.example.com < /dev/null 2>/dev/null | grep 'Verify return code'
Verify return code: 0 (ok)
Switching the client to TLS
Copy ca.pem (only the certificate, never ca-key.pem) to your home directory on the log server so you can transfer it:
sudo install -o "$USER" -m 0644 /etc/rsyslog.d/tls/ca.pem ~/ca.pem
Transfer ~/ca.pem to your home directory on the client, for example with scp, and install it on the client:
sudo install -d -o root -g syslog -m 0750 /etc/rsyslog.d/tls
sudo install -o root -g syslog -m 0644 ~/ca.pem /etc/rsyslog.d/tls/ca.pem
Replace /etc/rsyslog.d/90-forward.conf on the client:
sudo nano /etc/rsyslog.d/90-forward.conf
global(DefaultNetstreamDriverCAFile="/etc/rsyslog.d/tls/ca.pem")
action(type="omfwd"
target="logs.example.com"
port="6514"
protocol="tcp"
StreamDriver="gtls"
StreamDriverMode="1"
StreamDriverAuthMode="x509/name"
StreamDriverPermittedPeers="logs.example.com"
queue.type="LinkedList"
queue.filename="fwd_central"
queue.maxDiskSpace="1g"
queue.saveOnShutdown="on"
action.resumeRetryCount="-1")
With x509/name, the client only sends logs if the server presents a certificate signed by your CA whose name is logs.example.com. Validate, restart and repeat the test from Step 4:
sudo rsyslogd -N1
sudo systemctl restart rsyslog
logger -t rsyslog-test "TLS test from $(hostname)"
The message should appear in /var/log/remote/client01/rsyslog-test.log on the log server.
Step 6 - Rotating the collected logs
The default logrotate rules in Ubuntu do not know about /var/log/remote, so the files would grow forever. Create a rotation policy on the log server:
sudo nano /etc/logrotate.d/rsyslog-remote
/var/log/remote/*/*.log {
daily
rotate 30
missingok
notifempty
compress
delaycompress
sharedscripts
postrotate
/usr/lib/rsyslog/rsyslog-rotate
endscript
}
This keeps 30 days of logs per file, compresses older ones and tells rsyslog to reopen its files after rotation using the same helper that Ubuntu's own rsyslog rotation uses. Test the configuration without changing anything:
sudo logrotate -d /etc/logrotate.d/rsyslog-remote 2>&1 | head -n 20
The debug output lists each matched file and what would happen to it. logrotate runs daily through logrotate.timer, so no cron job is needed. Keep an eye on disk usage with:
sudo du -sh /var/log/remote/*
Troubleshooting
- Nothing arrives on the server. On the client, check
sudo journalctl -u rsyslog -n 50for connection errors and test the port withnc -vz your_log_server_ip 514(or 6514). On the server, confirm the listener withsudo ss -tlnp | grep rsyslogdand review the UFW rules. - The listener is up but no files are created. The
sysloguser probably cannot write to/var/log/remote. Check the owner withls -ld /var/log/remoteand look forPermission deniedinsudo journalctl -u rsyslog. - TLS errors such as
certificate invalidornot permitted to talk to peer. The name inStreamDriverPermittedPeersmust match the subject alternative name of the server certificate, and the client must resolve that name to the server. Inspect the certificate withsudo openssl x509 -in /etc/rsyslog.d/tls/server-cert.pem -noout -text | grep -A1 'Alternative'. - rsyslog cannot read the certificates. Look for AppArmor denials with
sudo journalctl -k | grep 'apparmor="DENIED"' | grep rsyslogand make sure the files live under/etc/rsyslog.d/with the permissions from Step 5. - The client queue keeps growing. Files in
/var/spool/rsyslog/namedfwd_central.*mean messages are waiting for the server. They are sent automatically once the connection works again.
Conclusion
You now have a central rsyslog server that stores logs per host and program, clients that forward their logs reliably over TCP with a disk queue, TLS encryption between them and a rotation policy for the stored files. From here you can add more clients by repeating Steps 3 and 5, search across all servers with grep and awk on /var/log/remote, or forward the central stream to a search engine such as OpenSearch or Loki when plain files are no longer enough.
