RustDesk Self-Hosted Remote Desktop

RustDesk is an open-source remote desktop solution that lets you run your own relay and signaling servers, keeping all remote access traffic within your infrastructure instead of routing it through third-party services like TeamViewer. With a self-hosted RustDesk server on Linux, you get full control over encryption keys, session logs, and access policies for enterprise deployments.

Prerequisites

  • Ubuntu 20.04+ or Debian 11+ (64-bit)
  • A public IP address or domain name pointing to your server
  • Root or sudo access
  • Minimum 1 vCPU, 1 GB RAM
  • Open ports: TCP 21115-21119, UDP 21116

Installing the RustDesk Server

Method 1: Docker Compose (recommended):

# Install Docker if not already installed
curl -fsSL https://get.docker.com | sudo bash

# Create the RustDesk server directory
sudo mkdir -p /opt/rustdesk
cd /opt/rustdesk

# Create Docker Compose file
sudo tee docker-compose.yml <<'EOF'
version: '3'
services:
  hbbs:
    image: rustdesk/rustdesk-server:latest
    command: hbbs
    volumes:
      - ./data:/root
    ports:
      - "21115:21115"
      - "21116:21116"
      - "21116:21116/udp"
      - "21118:21118"
    restart: unless-stopped
    network_mode: host

  hbbr:
    image: rustdesk/rustdesk-server:latest
    command: hbbr
    volumes:
      - ./data:/root
    ports:
      - "21117:21117"
      - "21119:21119"
    restart: unless-stopped
    network_mode: host
EOF

# Start the servers
sudo docker compose up -d

Method 2: Binary installation:

# Download the latest server package
RUSTDESK_VERSION="1.1.10"
wget "https://github.com/rustdesk/rustdesk-server/releases/download/${RUSTDESK_VERSION}/rustdesk-server-linux-amd64.zip"

# Extract
sudo apt install -y unzip
unzip rustdesk-server-linux-amd64.zip -d /opt/rustdesk/bin/

# Make executable
chmod +x /opt/rustdesk/bin/hbbs /opt/rustdesk/bin/hbbr

Create systemd services for binary install:

sudo tee /etc/systemd/system/rustdesk-hbbs.service <<'EOF'
[Unit]
Description=RustDesk Signal Server (hbbs)
After=network.target

[Service]
ExecStart=/opt/rustdesk/bin/hbbs -r your-server.example.com:21117
WorkingDirectory=/opt/rustdesk/data
Restart=always
RestartSec=5
User=root

[Install]
WantedBy=multi-user.target
EOF

sudo tee /etc/systemd/system/rustdesk-hbbr.service <<'EOF'
[Unit]
Description=RustDesk Relay Server (hbbr)
After=network.target

[Service]
ExecStart=/opt/rustdesk/bin/hbbr
WorkingDirectory=/opt/rustdesk/data
Restart=always
RestartSec=5
User=root

[Install]
WantedBy=multi-user.target
EOF

sudo systemctl daemon-reload
sudo systemctl enable --now rustdesk-hbbs rustdesk-hbbr

Server Configuration and Keys

On first start, RustDesk generates a key pair for encryption:

# View generated keys
ls /opt/rustdesk/data/
# id_ed25519      - private key (keep secret)
# id_ed25519.pub  - public key (share with clients)

# Get the public key to enter in clients
cat /opt/rustdesk/data/id_ed25519.pub

Clients need this public key to verify server identity and encrypt sessions.

Firewall Setup

# Ubuntu/Debian (UFW)
sudo ufw allow 21115/tcp   # NAT type test
sudo ufw allow 21116/tcp   # Signaling (TCP)
sudo ufw allow 21116/udp   # Signaling (UDP, hole punching)
sudo ufw allow 21117/tcp   # Relay server (hbbr)
sudo ufw allow 21118/tcp   # Web client (optional)
sudo ufw allow 21119/tcp   # Web client relay (optional)
sudo ufw reload

# CentOS/Rocky (firewalld)
for port in 21115 21116 21117 21118 21119; do
  sudo firewall-cmd --add-port=${port}/tcp --permanent
done
sudo firewall-cmd --add-port=21116/udp --permanent
sudo firewall-cmd --reload

Configuring Clients

Windows/macOS/Linux desktop client:

  1. Download the RustDesk client from https://rustdesk.com/
  2. Open Settings > Network
  3. Unlock settings (if locked)
  4. Set ID/Relay Server: your server's IP or domain (e.g., rustdesk.example.com)
  5. Set Key: paste the content of id_ed25519.pub from your server
  6. Click OK — the client will generate a unique ID

Pre-configure clients via command line (Linux):

# Configure server settings in the RustDesk config file
mkdir -p ~/.config/rustdesk/
cat > ~/.config/rustdesk/RustDesk2.toml <<EOF
rendezvous_server = 'rustdesk.example.com:21116'
nat_type = 1
serial = 0

[options]
custom-rendezvous-server = 'rustdesk.example.com'
key = 'YOUR_PUBLIC_KEY_HERE'
relay-server = 'rustdesk.example.com'
EOF

Unattended Access Setup

Enable unattended access so devices can be connected to without user interaction:

On the remote machine:

  1. Open RustDesk client
  2. Go to Settings > Security
  3. Set a permanent password under Password
  4. Enable Allow direct access and Run on startup
# On Linux, enable RustDesk to start as a service
sudo rustdesk --install-service
sudo systemctl enable --now rustdesk

# Set a fixed password via CLI
rustdesk --password "your-secure-password"

Connect using fixed password:

# Connect from CLI (Linux)
rustdesk --connect <remote-id> --password "your-secure-password"

Running Behind a Reverse Proxy

For HTTPS web client access, proxy port 21118 with Nginx:

# /etc/nginx/sites-available/rustdesk
server {
    listen 443 ssl;
    server_name rustdesk.example.com;

    ssl_certificate /etc/letsencrypt/live/rustdesk.example.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/rustdesk.example.com/privkey.pem;

    location / {
        proxy_pass http://127.0.0.1:21118;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
        proxy_set_header Host $host;
    }
}

sudo ln -s /etc/nginx/sites-available/rustdesk /etc/nginx/sites-enabled/
sudo nginx -t && sudo systemctl reload nginx

Troubleshooting

Client shows "Connection failed" or stays on "Connecting":

# Check both services are running
sudo docker compose ps    # or
sudo systemctl status rustdesk-hbbs rustdesk-hbbr

# Check port availability
sudo ss -tlnp | grep -E '2111[5-9]'

# View server logs
sudo docker compose logs -f hbbs
sudo docker compose logs -f hbbr

Client cannot register (no ID assigned):

# Verify TCP port 21116 is reachable from the client
# Test from a different machine
nc -zv your-server.example.com 21116

# Check the key matches exactly (no extra spaces or newlines)
cat /opt/rustdesk/data/id_ed25519.pub | wc -c

High latency sessions:

# Check if direct P2P connection is established (vs relay)
# In the RustDesk client, the connection type shows in the toolbar
# Direct = low latency, Relay = traffic goes through your server

# Ensure UDP 21116 is open for NAT hole punching
sudo ufw allow 21116/udp

Conclusion

RustDesk with a self-hosted server gives you a fully private remote desktop infrastructure with end-to-end encryption, eliminating dependency on commercial services while keeping latency low through direct P2P connections when possible. For enterprise deployments, combining fixed passwords, startup services, and reverse proxy access creates a robust remote management system that scales to hundreds of managed devices without per-seat licensing costs.