Sending Logs to Centralized Server with rsyslog

Introduction

Managing logs across multiple servers becomes increasingly challenging as infrastructure scales. Centralized logging solves this problem by aggregating logs from all servers into a single location, enabling unified search, analysis, correlation, and long-term storage. rsyslog is a powerful, high-performance logging system that excels at collecting and forwarding log data in distributed environments.

rsyslog has been the standard logging daemon on most Linux distributions for over a decade, offering reliable log transmission, flexible filtering, format transformation, and support for various protocols including TCP, UDP, and TLS-encrypted connections. Unlike heavyweight log shippers, rsyslog is lightweight, battle-tested, and already installed on most systems.

This comprehensive guide teaches you how to configure rsyslog for centralized logging, covering both client and server configurations. You'll learn how to set up secure log forwarding, implement log filtering and parsing, configure reliable delivery mechanisms, and integrate with modern logging stacks like Elasticsearch and Splunk. Whether you're managing a handful of servers or thousands, mastering rsyslog centralization is essential for effective log management.

Prerequisites

Before configuring centralized logging with rsyslog, ensure you have:

Multiple Linux servers (Ubuntu 20.04/22.04, Debian 10/11, CentOS 7/8, Rocky Linux 8/9)
Root or sudo access on all servers
Network connectivity between servers
Basic understanding of syslog protocol and log formats
One server designated as the centralized log server

System Requirements:

Log Server: 8+ GB RAM, 100+ GB disk (scales with log volume)
Client Servers: rsyslog installed (usually default)
Network: Stable connectivity, firewall configuration

Understanding rsyslog Architecture

Components

rsyslog consists of:

Input modules: Receive logs from various sources
Filter engine: Process and route logs based on rules
Output modules: Send logs to destinations
Queue system: Buffer logs for reliable delivery

Centralized Logging Topology

Client Server 1 (rsyslog client)
     |
     | TCP/UDP/TLS
     |
     v
Central Log Server (rsyslog server)
     |
     | (stores locally, forwards to analytics)
     |
     v
Elasticsearch / Splunk / File Storage

Installing and Verifying rsyslog

rsyslog is usually pre-installed on modern Linux distributions.

Verify Installation

# Check if rsyslog is installed
rsyslogd -v

# Check service status
sudo systemctl status rsyslog

# View rsyslog configuration
ls -l /etc/rsyslog.conf
ls -l /etc/rsyslog.d/

Install if Missing

On Ubuntu/Debian:

sudo apt update
sudo apt install rsyslog -y

On CentOS/Rocky Linux:

sudo yum install rsyslog -y

Start and enable rsyslog:

sudo systemctl enable rsyslog
sudo systemctl start rsyslog
sudo systemctl status rsyslog

Configuring Central Log Server

The central server receives logs from all client servers and stores them centrally.

Basic Server Configuration

Edit rsyslog configuration:

sudo nano /etc/rsyslog.conf

Enable UDP reception (port 514):

# Uncomment these lines to enable UDP syslog reception
module(load="imudp")
input(type="imudp" port="514")

Enable TCP reception (port 514) - Recommended for reliability:

# Uncomment these lines to enable TCP syslog reception
module(load="imtcp")
input(type="imtcp" port="514")

Create template for organizing logs by hostname:

# Add at the end of /etc/rsyslog.conf or create /etc/rsyslog.d/30-remote.conf
sudo nano /etc/rsyslog.d/30-remote.conf

# Template to organize logs by hostname
$template RemoteLogs,"/var/log/remote/%HOSTNAME%/%PROGRAMNAME%.log"

# Store remote logs using template
*.* ?RemoteLogs

# Also stop processing after storing (comment if you want duplicate to /var/log/syslog)
& stop

Restart rsyslog:

sudo systemctl restart rsyslog
sudo systemctl status rsyslog

Advanced Server Configuration

More sophisticated log organization:

sudo nano /etc/rsyslog.d/30-remote.conf

# Dynamic file template with date
$template DailyPerHostLogs,"/var/log/remote/%HOSTNAME%/%$YEAR%-%$MONTH%-%$DAY%/%PROGRAMNAME%.log"

# Template for emergency logs
$template EmergencyLogs,"/var/log/remote/%HOSTNAME%/emergency.log"

# Route logs based on severity
if $fromsrv-ip startswith '192.168.1.' then {
    # Emergency and critical logs to separate file
    if $syslogseverity <= 2 then {
        action(type="omfile" dynaFile="EmergencyLogs")
    }

    # All logs to daily organized files
    action(type="omfile" dynaFile="DailyPerHostLogs")

    # Stop further processing for remote logs
    stop
}

Filter by facility:

# Separate authentication logs
auth,authpriv.*   /var/log/remote/%HOSTNAME%/auth.log

# Separate cron logs
cron.*            /var/log/remote/%HOSTNAME%/cron.log

# Separate mail logs
mail.*            /var/log/remote/%HOSTNAME%/mail.log

Create Log Directories

# Create remote log directory
sudo mkdir -p /var/log/remote

# Set appropriate permissions
sudo chmod 755 /var/log/remote

# Create subdirectories will be created automatically by rsyslog

Firewall Configuration

Allow incoming syslog traffic:

UFW (Ubuntu/Debian):

# Allow UDP syslog (if enabled)
sudo ufw allow 514/udp

# Allow TCP syslog (recommended)
sudo ufw allow 514/tcp

# Allow from specific subnet only
sudo ufw allow from 192.168.1.0/24 to any port 514 proto tcp

firewalld (CentOS/Rocky Linux):

# Allow syslog service
sudo firewall-cmd --permanent --add-service=syslog

# Or specific ports
sudo firewall-cmd --permanent --add-port=514/tcp
sudo firewall-cmd --permanent --add-port=514/udp

# Allow from specific source
sudo firewall-cmd --permanent --add-rich-rule='rule family="ipv4" source address="192.168.1.0/24" port port="514" protocol="tcp" accept'

# Reload firewall
sudo firewall-cmd --reload

iptables:

# Allow TCP syslog
sudo iptables -A INPUT -p tcp --dport 514 -j ACCEPT

# Allow UDP syslog
sudo iptables -A INPUT -p udp --dport 514 -j ACCEPT

# Save rules
sudo netfilter-persistent save

Configuring Client Servers

Client servers forward their logs to the central server.

Basic Client Configuration

Edit rsyslog configuration:

sudo nano /etc/rsyslog.d/50-central-logging.conf

Forward all logs via UDP:

# Forward all logs to central server (UDP)
*.* @192.168.1.100:514

Forward via TCP (recommended for reliability):

# Forward all logs to central server (TCP)
*.* @@192.168.1.100:514

Forward with queue for reliability:

# Forward logs with disk-assisted queue
*.* action(type="omfwd"
           target="192.168.1.100"
           port="514"
           protocol="tcp"
           queue.filename="fwdRule1"
           queue.maxdiskspace="1g"
           queue.saveonshutdown="on"
           queue.type="LinkedList"
           resumeRetryCount="-1")

Restart rsyslog:

sudo systemctl restart rsyslog
sudo systemctl status rsyslog

Selective Log Forwarding

Forward only specific facilities:

# Forward only authentication logs
auth,authpriv.* @@192.168.1.100:514

# Forward only critical messages
*.crit @@192.168.1.100:514

# Forward everything except debug
*.info;*.!debug @@192.168.1.100:514

Forward with local copy:

# Forward to central server
*.* @@192.168.1.100:514

# Also keep local copy
*.* /var/log/syslog

Exclude certain logs from forwarding:

# Don't forward debug messages
*.info;*.!debug @@192.168.1.100:514

# Don't forward cron messages
*.*;cron.none @@192.168.1.100:514

Application-Specific Forwarding

Forward Apache logs:

sudo nano /etc/rsyslog.d/apache-forward.conf

# Load file input module
module(load="imfile" PollingInterval="10")

# Apache access log
input(type="imfile"
      File="/var/log/apache2/access.log"
      Tag="apache-access"
      Severity="info"
      Facility="local6")

# Apache error log
input(type="imfile"
      File="/var/log/apache2/error.log"
      Tag="apache-error"
      Severity="error"
      Facility="local6")

# Forward to central server
local6.* @@192.168.1.100:514

Forward Nginx logs:

sudo nano /etc/rsyslog.d/nginx-forward.conf

module(load="imfile")

input(type="imfile"
      File="/var/log/nginx/access.log"
      Tag="nginx-access"
      Severity="info"
      Facility="local7")

input(type="imfile"
      File="/var/log/nginx/error.log"
      Tag="nginx-error"
      Severity="error"
      Facility="local7")

local7.* @@192.168.1.100:514

Forward application logs:

sudo nano /etc/rsyslog.d/app-forward.conf

module(load="imfile")

input(type="imfile"
      File="/var/log/myapp/app.log"
      Tag="myapp"
      Severity="info"
      Facility="local5"
      reopenOnTruncate="on")

local5.* action(type="omfwd"
                target="192.168.1.100"
                port="514"
                protocol="tcp")

Secure Log Transmission with TLS

For production environments, encrypt log transmission to protect sensitive data.

Server TLS Configuration

Generate TLS certificates:

# Create directory for certificates
sudo mkdir -p /etc/rsyslog.d/certs

# Generate CA certificate
sudo openssl genrsa -out /etc/rsyslog.d/certs/ca-key.pem 2048
sudo openssl req -new -x509 -key /etc/rsyslog.d/certs/ca-key.pem \
    -out /etc/rsyslog.d/certs/ca-cert.pem -days 3650 \
    -subj "/CN=RSyslog-CA"

# Generate server certificate
sudo openssl genrsa -out /etc/rsyslog.d/certs/server-key.pem 2048
sudo openssl req -new -key /etc/rsyslog.d/certs/server-key.pem \
    -out /etc/rsyslog.d/certs/server-req.pem \
    -subj "/CN=log-server.example.com"

# Sign server certificate
sudo openssl x509 -req -in /etc/rsyslog.d/certs/server-req.pem \
    -CA /etc/rsyslog.d/certs/ca-cert.pem \
    -CAkey /etc/rsyslog.d/certs/ca-key.pem \
    -CAcreateserial \
    -out /etc/rsyslog.d/certs/server-cert.pem -days 3650

# Set permissions
sudo chmod 600 /etc/rsyslog.d/certs/*key.pem

Configure rsyslog for TLS:

sudo nano /etc/rsyslog.d/30-remote-tls.conf

# Load network stream driver
module(load="imtcp"
       StreamDriver.Name="gtls"
       StreamDriver.Mode="1"
       StreamDriver.Authmode="x509/name"
       PermittedPeer=["*.example.com"])

# Set certificate files
global(
    DefaultNetstreamDriver="gtls"
    DefaultNetstreamDriverCAFile="/etc/rsyslog.d/certs/ca-cert.pem"
    DefaultNetstreamDriverCertFile="/etc/rsyslog.d/certs/server-cert.pem"
    DefaultNetstreamDriverKeyFile="/etc/rsyslog.d/certs/server-key.pem"
)

# TLS input on port 6514
input(type="imtcp" port="6514")

# Log template
$template RemoteLogs,"/var/log/remote/%HOSTNAME%/%PROGRAMNAME%.log"
*.* ?RemoteLogs
& stop

Restart rsyslog:

sudo systemctl restart rsyslog

Client TLS Configuration

Copy CA certificate to clients:

# On server
sudo cat /etc/rsyslog.d/certs/ca-cert.pem

# On client, create directory and save certificate
sudo mkdir -p /etc/rsyslog.d/certs
sudo nano /etc/rsyslog.d/certs/ca-cert.pem
# Paste certificate content

Configure client for TLS:

sudo nano /etc/rsyslog.d/50-central-logging-tls.conf

# Load network stream driver
module(load="omfwd")

# Global TLS settings
global(
    DefaultNetstreamDriver="gtls"
    DefaultNetstreamDriverCAFile="/etc/rsyslog.d/certs/ca-cert.pem"
)

# Forward logs via TLS
*.* action(type="omfwd"
           target="log-server.example.com"
           port="6514"
           protocol="tcp"
           StreamDriver="gtls"
           StreamDriverMode="1"
           StreamDriverAuthMode="x509/name"
           StreamDriverPermittedPeers="log-server.example.com"
           queue.filename="fwdRule1"
           queue.maxdiskspace="1g"
           queue.saveonshutdown="on"
           queue.type="LinkedList"
           resumeRetryCount="-1")

Restart rsyslog:

sudo systemctl restart rsyslog

Testing Log Forwarding

Generate Test Logs

On client server:

# Generate test syslog message
logger -t test-app "This is a test message from $(hostname)"

# Generate with specific priority
logger -p local0.info "Test message with facility local0"

# Generate multiple test messages
for i in {1..10}; do
    logger -t test-app "Test message $i from $(hostname)"
    sleep 1
done

Verify on Central Server

# Check if logs are arriving
sudo tail -f /var/log/remote/client-hostname/test-app.log

# Check all remote logs
sudo tail -f /var/log/remote/*/*.log

# Search for specific message
sudo grep "test message" /var/log/remote/*/*

Monitor rsyslog Status

On client:

# Check rsyslog status
sudo systemctl status rsyslog

# View rsyslog stats
sudo rsyslogd -i /var/run/rsyslogd.pid -x

# Check for errors
sudo journalctl -u rsyslog | grep -i error

# View queue status
sudo ls -lh /var/spool/rsyslog/

On server:

# Monitor incoming connections
sudo netstat -an | grep :514

# Or using ss
sudo ss -tln | grep :514

# Check disk usage
du -sh /var/log/remote/

# Count incoming logs
sudo tail -f /var/log/remote/*/*.log | wc -l

Log Rotation for Central Server

With centralized logging, implement proper log rotation to manage disk space.

Configure logrotate

sudo nano /etc/logrotate.d/rsyslog-remote

/var/log/remote/*/*.log
{
    daily
    rotate 30
    missingok
    notifempty
    compress
    delaycompress
    sharedscripts
    postrotate
        /bin/kill -HUP $(cat /var/run/rsyslogd.pid 2> /dev/null) 2> /dev/null || true
    endscript
}

# Emergency logs - keep longer
/var/log/remote/*/emergency.log
{
    weekly
    rotate 52
    missingok
    notifempty
    compress
    delaycompress
    sharedscripts
}

Automated Cleanup Script

sudo nano /usr/local/bin/cleanup-old-logs.sh

#!/bin/bash
# cleanup-old-logs.sh - Remove logs older than 90 days

LOG_DIR="/var/log/remote"
RETENTION_DAYS=90

echo "Cleaning up logs older than $RETENTION_DAYS days in $LOG_DIR"

# Find and delete old log files
find "$LOG_DIR" -type f -name "*.log.gz" -mtime +$RETENTION_DAYS -delete

# Find and delete old log files
find "$LOG_DIR" -type f -name "*.log" -mtime +$RETENTION_DAYS -delete

# Remove empty directories
find "$LOG_DIR" -type d -empty -delete

echo "Cleanup completed"

# Make executable
sudo chmod +x /usr/local/bin/cleanup-old-logs.sh

# Add to cron (run weekly)
sudo crontab -e
# Add: 0 2 * * 0 /usr/local/bin/cleanup-old-logs.sh

Integration with Other Systems

Forward to Elasticsearch

sudo nano /etc/rsyslog.d/elasticsearch.conf

# Load Elasticsearch output module
module(load="omelasticsearch")

# Template for Elasticsearch
template(name="elasticsearch-template"
         type="list") {
    constant(value="{\"timestamp\":\"")     property(name="timereported" dateFormat="rfc3339")
    constant(value="\",\"host\":\"")        property(name="hostname")
    constant(value="\",\"severity\":\"")    property(name="syslogseverity-text")
    constant(value="\",\"facility\":\"")    property(name="syslogfacility-text")
    constant(value="\",\"program\":\"")     property(name="programname")
    constant(value="\",\"message\":\"")     property(name="msg" format="json")
    constant(value="\"}")
}

# Send to Elasticsearch
*.* action(type="omelasticsearch"
           server="localhost"
           serverport="9200"
           template="elasticsearch-template"
           searchIndex="syslog-index"
           searchType="events"
           bulkmode="on"
           queue.type="linkedlist"
           queue.size="5000"
           queue.dequeuebatchsize="300"
           action.resumeretrycount="-1")

Forward to Remote Syslog Server (Relay)

# Central server can also relay to another server
sudo nano /etc/rsyslog.d/relay.conf

# Relay to backup server
*.* @@backup-log-server.example.com:514

# Or relay to cloud service
*.* @@logs.papertrailapp.com:12345

Performance Tuning

Optimize Server Performance

sudo nano /etc/rsyslog.d/tuning.conf

# Increase message queue size
$MainMsgQueueSize 100000

# Use disk-assisted queue
$MainMsgQueueType LinkedList
$MainMsgQueueFileName mainqueue
$WorkDirectory /var/spool/rsyslog

# Increase number of threads
$MainMsgQueueWorkerThreads 4

# Batch writes for better performance
$OMFileAsyncWriting on
$OMFileFlushInterval 10

Client Performance

# Use buffered sending
*.* action(type="omfwd"
           target="192.168.1.100"
           port="514"
           protocol="tcp"
           queue.type="LinkedList"
           queue.filename="remote"
           queue.maxdiskspace="1g"
           queue.size="10000"
           queue.dequeuebatchsize="100")

Troubleshooting

Debug Mode

# Run rsyslog in debug mode
sudo rsyslogd -N1 -d

# Or edit config to enable debug
sudo nano /etc/rsyslog.conf
# Add: $DebugLevel 2
# Add: $DebugFile /var/log/rsyslog-debug.log

Common Issues

Issue 1: Logs not forwarding

# Check if rsyslog is running
sudo systemctl status rsyslog

# Test network connectivity
telnet 192.168.1.100 514

# Check firewall
sudo iptables -L -n | grep 514

# Verify configuration syntax
sudo rsyslogd -N1

# Check for errors
sudo journalctl -u rsyslog | grep -i error

Issue 2: Logs not appearing on server

# Verify server is listening
sudo ss -tln | grep 514

# Check if logs directory exists
ls -ld /var/log/remote

# Verify permissions
ls -l /var/log/remote

# Check SELinux (if applicable)
sudo sestatus
sudo setenforce 0  # Temporarily disable to test

Issue 3: TLS connection failures

# Verify certificates
openssl x509 -in /etc/rsyslog.d/certs/server-cert.pem -text -noout

# Test TLS connection
openssl s_client -connect log-server.example.com:6514

# Check certificate permissions
ls -l /etc/rsyslog.d/certs/

# View detailed errors
sudo rsyslogd -N1 -d 2>&1 | grep -i tls

Conclusion

Centralized logging with rsyslog provides a robust, scalable solution for log aggregation across distributed infrastructure. By implementing the configurations in this guide, you've created a reliable logging system that ensures all server logs are collected, secured, and accessible from a central location.

Key takeaways:

Centralization - Aggregate logs from multiple servers for unified analysis
Reliability - Use TCP with queuing for guaranteed log delivery
Security - Implement TLS encryption for sensitive log data
Flexibility - Filter and route logs based on facility, severity, and content
Integration - Forward logs to modern analytics platforms

Best practices:

Use TCP with queuing for production environments
Implement TLS for log transmission security
Configure proper log rotation and retention
Monitor disk space on central log server
Test failover scenarios and queue recovery
Document your logging infrastructure
Implement access controls on log files
Regular backup of critical logs

rsyslog's lightweight footprint, proven reliability, and extensive feature set make it ideal for centralized logging. Whether you're forwarding to local storage, Elasticsearch, Splunk, or cloud logging services, rsyslog provides the flexible foundation needed for comprehensive log management in modern infrastructure.

Sending Logs to Centralized Server with rsyslog

Sending Logs to Centralized Server with rsyslog

Introduction

Prerequisites

Understanding rsyslog Architecture

Components

Centralized Logging Topology

Installing and Verifying rsyslog

Verify Installation

Install if Missing

Configuring Central Log Server

Basic Server Configuration

Advanced Server Configuration

Create Log Directories

Firewall Configuration

Configuring Client Servers

Basic Client Configuration

Selective Log Forwarding

Application-Specific Forwarding

Secure Log Transmission with TLS

Server TLS Configuration

Client TLS Configuration

Testing Log Forwarding

Generate Test Logs

Verify on Central Server

Monitor rsyslog Status

Log Rotation for Central Server

Configure logrotate

Automated Cleanup Script

Integration with Other Systems

Forward to Elasticsearch

Forward to Remote Syslog Server (Relay)

Performance Tuning

Optimize Server Performance

Client Performance

Troubleshooting

Debug Mode

Common Issues

Conclusion

Latest Video

Get $20 Free Credit