mTLS Mutual TLS Configuration

Mutual TLS (mTLS) extends standard TLS by requiring both client and server to authenticate each other using certificates. Unlike traditional HTTPS where only the server proves its identity, mTLS provides bidirectional authentication, ensuring that only trusted clients access protected services. This guide covers CA setup, client certificate creation, configuration across Nginx/HAProxy/Apache, revocation mechanisms, certificate rotation, and troubleshooting strategies.

Table of Contents

1. mTLS Overview
2. Certificate Authority Setup
3. Server Certificate Creation
4. Client Certificate Creation
5. Nginx mTLS Configuration
6. HAProxy mTLS Configuration
7. Apache mTLS Configuration
8. Certificate Revocation
9. Certificate Rotation
10. mTLS Testing
11. Troubleshooting

mTLS Overview

mTLS authentication flow:

1. Client connects and sends its certificate
2. Server validates client certificate against CA
3. Server presents its certificate
4. Client validates server certificate
5. Both parties verify Certificate Chain

Benefits:

• Prevents unauthorized client access
• Detects compromised client credentials
• Eliminates password authentication need
• Suitable for service-to-service authentication
• No shared secrets to steal

Use cases:

• Internal API communication
• Service mesh (Istio, Linkerd)
• Microservices authentication
• Admin/dashboard access
• IoT device authentication

Certificate Authority Setup

Create a private CA for internal certificates:

# Create CA directory
mkdir -p ~/ca/{private,certs,csr}
cd ~/ca

# Generate CA private key (4096-bit RSA)
openssl genrsa -out private/ca.key 4096

# Generate CA certificate (valid 10 years)
openssl req -new -x509 -days 3650 -key private/ca.key -out certs/ca.crt \
  -subj "/C=US/ST=State/L=City/O=Organization/CN=Internal-CA"

Verify CA certificate:

openssl x509 -in certs/ca.crt -text -noout

Create CA configuration for automated signing:

cat > ~/ca/ca.conf <<'EOF'
[ ca ]
default_ca = CA_default

[ CA_default ]
dir = /root/ca
certs = $dir/certs
crl_dir = $dir/crl
new_certs_dir = $dir/certs
database = $dir/index.txt
serial = $dir/serial
RANDFILE = $dir/private/.rand

default_crl_days = 30
default_crl_extensions = crl_ext
default_days = 365
default_md = sha256
name_opt = ca_default
cert_opt = ca_default
preserve = no
policy = policy_strict
unique_subject = no

[ policy_strict ]
countryName = match
stateOrProvinceName = match
organizationName = match
organizationalUnitName = optional
commonName = supplied
emailAddress = optional

[ policy_loose ]
countryName = optional
stateOrProvinceName = optional
localityName = optional
organizationName = optional
organizationalUnitName = optional
commonName = supplied
emailAddress = optional

[ req ]
default_bits = 4096
distinguished_name = req_distinguished_name
string_mask = utf8only
default_md = sha256
x509_extensions = v3_ca

[ req_distinguished_name ]
countryName = Country Name
stateOrProvinceName = State or Province Name
localityName = Locality Name
organizationName = Organization Name
organizationalUnitName = Organizational Unit Name
commonName = Common Name
emailAddress = Email Address

[ v3_ca ]
subjectKeyIdentifier = hash
authorityKeyIdentifier = keyid:always,issuer
basicConstraints = critical, CA:true
keyUsage = critical, digitalSignature, cRLSign, keyCertSign
EOF

Initialize CA database:

cd ~/ca
touch index.txt
echo 1000 > serial

Server Certificate Creation

Create server certificate signed by CA:

# Generate server private key
openssl genrsa -out ~/ca/private/server.key 4096

# Create server CSR (Certificate Signing Request)
openssl req -new -key ~/ca/private/server.key -out ~/ca/csr/server.csr \
  -subj "/C=US/ST=State/L=City/O=Organization/CN=api.example.com"

# Sign server certificate with CA
openssl ca -config ~/ca/ca.conf -extensions v3_ca -days 365 \
  -notext -md sha256 -in ~/ca/csr/server.csr \
  -out ~/ca/certs/server.crt

# Create combined certificate and key
cat ~/ca/certs/server.crt ~/ca/private/server.key > ~/ca/certs/server.pem

Add certificate extensions for mTLS:

# Create server config with SANs
cat > ~/ca/server.conf <<'EOF'
[ req ]
default_bits = 4096
distinguished_name = req_distinguished_name
req_extensions = v3_req

[ req_distinguished_name ]
countryName = Country Name
stateOrProvinceName = State or Province Name
commonName = Common Name

[ v3_req ]
subjectAltName = @alt_names

[ alt_names ]
DNS.1 = api.example.com
DNS.2 = *.api.example.com
DNS.3 = api.internal
IP.1 = 192.168.1.100
IP.2 = 10.0.0.1
EOF

# Generate CSR with extensions
openssl req -new -key ~/ca/private/server.key -out ~/ca/csr/server.csr \
  -config ~/ca/server.conf \
  -subj "/C=US/ST=State/L=City/O=Organization/CN=api.example.com"

Client Certificate Creation

Create client certificates for authentication:

# Generate client private key
openssl genrsa -out ~/ca/private/client1.key 4096

# Create client CSR
openssl req -new -key ~/ca/private/client1.key -out ~/ca/csr/client1.csr \
  -subj "/C=US/ST=State/L=City/O=Organization/CN=app-service-1"

# Sign client certificate
openssl ca -config ~/ca/ca.conf -days 365 -notext -md sha256 \
  -in ~/ca/csr/client1.csr -out ~/ca/certs/client1.crt

# Create PKCS12 format (for some applications)
openssl pkcs12 -export -in ~/ca/certs/client1.crt \
  -inkey ~/ca/private/client1.key -out ~/ca/certs/client1.p12 \
  -name "App Service 1" -passout pass:password

Create multiple client certificates:

# Batch create certificates
for client in app1 app2 app3 app4; do
  openssl genrsa -out ~/ca/private/$client.key 4096
  openssl req -new -key ~/ca/private/$client.key -out ~/ca/csr/$client.csr \
    -subj "/C=US/ST=State/L=City/O=Organization/CN=$client"
  openssl ca -config ~/ca/ca.conf -days 365 -notext -md sha256 \
    -in ~/ca/csr/$client.csr -out ~/ca/certs/$client.crt
done

Nginx mTLS Configuration

Configure Nginx to require and validate client certificates:

upstream backend {
    server 192.168.1.100:8000;
    server 192.168.1.101:8000;
    keepalive 32;
}

server {
    listen 443 ssl http2;
    server_name api.example.com;
    
    # Server certificate and key
    ssl_certificate /etc/nginx/ssl/server.crt;
    ssl_certificate_key /etc/nginx/ssl/server.key;
    
    # Client certificate validation
    ssl_client_certificate /etc/nginx/ssl/ca.crt;
    ssl_verify_client on;
    ssl_verify_depth 2;
    ssl_client_session_cache shared:SSL:10m;
    
    # TLS Configuration
    ssl_protocols TLSv1.2 TLSv1.3;
    ssl_ciphers HIGH:!aNULL:!MD5;
    ssl_session_cache shared:SSL:10m;
    ssl_session_timeout 10m;
    
    # Pass client certificate to backend
    proxy_set_header SSL-Client-Cert $ssl_client_cert;
    proxy_set_header SSL-Client-Verify $ssl_client_verify;
    proxy_set_header SSL-Client-Subject $ssl_client_s_dn;
    
    location / {
        # Optional: Allow access only to valid certs
        if ($ssl_client_verify != SUCCESS) {
            return 403;
        }
        
        proxy_pass http://backend;
        proxy_http_version 1.1;
        proxy_set_header Connection "";
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    }
}

# Optional client certificate section
server {
    listen 8443 ssl;
    server_name api.example.com;
    
    # Require valid client cert for admin
    ssl_client_certificate /etc/nginx/ssl/ca.crt;
    ssl_verify_client optional;
    
    location /admin {
        if ($ssl_client_verify != SUCCESS) {
            return 401;
        }
        # Admin only content
    }
}

HAProxy mTLS Configuration

Configure HAProxy for mTLS:

global
    log stdout local0
    tune.ssl.default-dh-param 2048

frontend https_in
    bind *:443 ssl crt /etc/haproxy/ssl/server.pem \
        ca-file /etc/haproxy/ssl/ca.crt \
        verify optional
    
    mode http
    option httpclose
    option forwardfor
    
    # Pass client certificate info
    http-request set-header X-SSL-Client-Cert %{+Q}[ssl_c_der,hex]
    http-request set-header X-SSL-Client-Verify %[ssl_c_verify]
    http-request set-header X-SSL-Client-Subject %{+Q}[ssl_c_s_dn]
    http-request set-header X-SSL-Client-CN %{+Q}[ssl_c_s_dn(cn)]
    
    # Deny requests without valid client cert
    http-request deny if !{ ssl_c_verify eq 0 }
    
    default_backend backend_servers

backend backend_servers
    balance roundrobin
    mode http
    server srv1 192.168.1.100:8000 check
    server srv2 192.168.1.101:8000 check

Advanced HAProxy with certificate validation:

frontend https_in
    bind *:443 ssl crt /etc/haproxy/ssl/server.pem \
        ca-file /etc/haproxy/ssl/ca.crt \
        verify required
    
    http-request set-header X-Client-CN %{+Q}[ssl_c_s_dn(cn)]
    
    acl allowed_clients src -f /etc/haproxy/allowed_clients.txt
    acl client_cert_valid ssl_c_verify eq 0
    acl cert_cn_admin ssl_c_s_dn(cn) -i admin
    acl cert_cn_readonly ssl_c_s_dn(cn) -i readonly
    
    use_backend backend_admin if client_cert_valid cert_cn_admin
    use_backend backend_readonly if client_cert_valid cert_cn_readonly
    default_backend backend_servers

backend backend_admin
    balance roundrobin
    server srv1 192.168.1.100:8001 check

backend backend_readonly
    balance roundrobin
    server srv2 192.168.1.101:8001 check

backend backend_servers
    balance roundrobin
    server srv1 192.168.1.100:8000 check

Apache mTLS Configuration

Configure Apache for mTLS:

<VirtualHost *:443>
    ServerName api.example.com
    
    # Server certificate
    SSLCertificateFile /etc/apache2/ssl/server.crt
    SSLCertificateKeyFile /etc/apache2/ssl/server.key
    SSLCertificateChainFile /etc/apache2/ssl/ca.crt
    
    # Client certificate validation
    SSLVerifyClient require
    SSLVerifyDepth 2
    SSLCACertificateFile /etc/apache2/ssl/ca.crt
    
    # TLS Configuration
    SSLProtocol TLSv1.2 TLSv1.3
    SSLCipherSuite HIGH:!aNULL:!MD5
    SSLHonorCipherOrder on
    
    # Pass client info to backend
    SSLOptions +StdEnvVars
    RequestHeader set X-SSL-Client-Cert %{SSL_CLIENT_CERT}e
    RequestHeader set X-SSL-Client-Verify %{SSL_CLIENT_VERIFY}e
    RequestHeader set X-SSL-Client-Subject %{SSL_CLIENT_S_DN}e
    RequestHeader set X-SSL-Client-CN %{SSL_CLIENT_S_DN_CN}e
    
    <Location />
        ProxyPass http://192.168.1.100:8000/
        ProxyPassReverse http://192.168.1.100:8000/
    </Location>
</VirtualHost>

# Optional: Different policies for different paths
<VirtualHost *:443>
    ServerName api.example.com
    
    # Admin requires certificate
    <Location /admin>
        SSLVerifyClient require
    </Location>
    
    # Public API optional certificate
    <Location /public>
        SSLVerifyClient optional
    </Location>
</VirtualHost>

Certificate Revocation

Implement CRL (Certificate Revocation List) or OCSP:

CRL Setup

# Create CRL file
openssl ca -config ~/ca/ca.conf -gencrl -out ~/ca/crl/ca.crl

# Revoke a client certificate
openssl ca -config ~/ca/ca.conf -revoke ~/ca/certs/client1.crt

# Update CRL
openssl ca -config ~/ca/ca.conf -gencrl -out ~/ca/crl/ca.crl

# Verify revocation
openssl crl -in ~/ca/crl/ca.crl -text -noout

Configure Nginx with CRL:

ssl_crl /etc/nginx/ssl/ca.crl;
ssl_verify_client on;
ssl_verify_depth 2;

OCSP Stapling

Enable OCSP stapling for client certificates:

ssl_stapling on;
ssl_stapling_verify on;
ssl_stapling_responder "http://ocsp.example.com";

Certificate Rotation

Implement automated certificate rotation:

#!/bin/bash
# Certificate rotation script

CERT_DIR="/etc/nginx/ssl"
CA_DIR="${HOME}/ca"

# Rotate certificates
for app in client1 client2 client3; do
  # Check if expiration is within 30 days
  EXPIRATION=$(openssl x509 -in ${CA_DIR}/certs/${app}.crt -noout -enddate | cut -d= -f2)
  DAYS_LEFT=$(( ($(date -d "$EXPIRATION" +%s) - $(date +%s)) / 86400 ))
  
  if [ "$DAYS_LEFT" -lt 30 ]; then
    echo "Rotating certificate for $app (expires in $DAYS_LEFT days)"
    
    # Generate new key and CSR
    openssl genrsa -out ${CA_DIR}/private/${app}-new.key 4096
    openssl req -new -key ${CA_DIR}/private/${app}-new.key \
      -out ${CA_DIR}/csr/${app}-new.csr \
      -subj "/C=US/ST=State/L=City/O=Organization/CN=$app"
    
    # Sign new certificate
    openssl ca -config ${CA_DIR}/ca.conf -days 365 -notext -md sha256 \
      -in ${CA_DIR}/csr/${app}-new.csr \
      -out ${CA_DIR}/certs/${app}-new.crt
    
    # Backup old cert and key
    mv ${CERT_DIR}/${app}.key ${CERT_DIR}/${app}.key.bak
    mv ${CERT_DIR}/${app}.crt ${CERT_DIR}/${app}.crt.bak
    
    # Install new cert and key
    cp ${CA_DIR}/private/${app}-new.key ${CERT_DIR}/${app}.key
    cp ${CA_DIR}/certs/${app}-new.crt ${CERT_DIR}/${app}.crt
    
    # Reload web server
    systemctl reload nginx
  fi
done

Schedule rotation:

# Add to crontab
0 2 * * 0 /usr/local/bin/rotate-certs.sh

mTLS Testing

Test mTLS configuration with curl:

# Test without client certificate (should fail)
curl -v https://api.example.com/ 2>&1 | grep -i certificate

# Test with client certificate
curl -v --cert ~/ca/certs/client1.crt \
  --key ~/ca/private/client1.key \
  --cacert ~/ca/certs/ca.crt \
  https://api.example.com/

# Verify certificate chain
openssl s_client -connect api.example.com:443 \
  -cert ~/ca/certs/client1.crt \
  -key ~/ca/private/client1.key \
  -CAfile ~/ca/certs/ca.crt

Test with different clients:

# Test with client1
curl --cert ~/ca/certs/client1.crt --key ~/ca/private/client1.key \
  --cacert ~/ca/certs/ca.crt https://api.example.com/api/protected

# Test with client2
curl --cert ~/ca/certs/client2.crt --key ~/ca/private/client2.key \
  --cacert ~/ca/certs/ca.crt https://api.example.com/api/protected

Python testing:

import requests
from requests.auth import HTTPClientCertAuth

response = requests.get(
    'https://api.example.com/api/protected',
    cert=('/root/ca/certs/client1.crt', '/root/ca/private/client1.key'),
    verify='/root/ca/certs/ca.crt'
)
print(response.status_code)

Troubleshooting

Verify certificate details:

# Check server certificate
openssl x509 -in /etc/nginx/ssl/server.crt -text -noout

# Check client certificate
openssl x509 -in ~/ca/certs/client1.crt -text -noout

# Verify certificate chain
openssl verify -CAfile ~/ca/certs/ca.crt ~/ca/certs/client1.crt

Test certificate validity:

# Check expiration
openssl x509 -in /etc/nginx/ssl/server.crt -noout -dates

# Check if revoked
openssl crl -in ~/ca/crl/ca.crl -text -noout | grep -i client1

Debug connection issues:

# Verbose SSL handshake
openssl s_client -connect api.example.com:443 -showcerts \
  -cert ~/ca/certs/client1.crt -key ~/ca/private/client1.key

# Check Nginx logs
tail -f /var/log/nginx/error.log | grep -i ssl

# tcpdump TLS handshake
sudo tcpdump -i eth0 -A "tcp port 443" | head -100

Conclusion

mTLS provides robust mutual authentication between clients and servers, essential for securing internal service communication. Proper CA setup, certificate generation, configuration across proxies, and lifecycle management ensure secure, maintainable mTLS deployments. Regular certificate rotation, revocation monitoring, and thorough testing maintain security posture and prevent authentication failures.