SSH Connectivity Issues: Diagnostics

Introduction

SSH (Secure Shell) is the primary protocol for secure remote server administration, making SSH connectivity issues one of the most frustrating problems system administrators encounter. When you cannot connect to your server via SSH, you're effectively locked out of remote management capabilities, which can severely impact operations and productivity.

This comprehensive guide provides a systematic approach to diagnosing and resolving SSH connectivity issues. Whether you're dealing with connection timeouts, authentication failures, or configuration problems, this guide will walk you through methodical troubleshooting steps using command-line diagnostic tools.

Understanding SSH connectivity problems is essential for DevOps engineers, system administrators, and anyone managing Linux servers. This guide covers everything from basic connectivity testing to advanced SSH debugging techniques, helping you quickly identify and resolve issues.

Understanding SSH Connection Process

SSH Connection Flow

To effectively troubleshoot SSH issues, understand the connection process:

  1. DNS Resolution: Client resolves server hostname to IP address
  2. Network Connectivity: TCP connection established on port 22 (default)
  3. SSH Protocol Negotiation: Client and server agree on protocol version
  4. Key Exchange: Cryptographic keys exchanged for encryption
  5. Authentication: User credentials verified (password or key-based)
  6. Session Establishment: Shell session created for user

Problems can occur at any of these stages, requiring different diagnostic approaches.

Common SSH Error Messages

Understanding error messages helps identify the problem quickly:

  • "Connection timed out": Network unreachable, firewall blocking, wrong IP/port
  • "Connection refused": SSH service not running or wrong port
  • "Permission denied": Authentication failure, wrong credentials or keys
  • "Host key verification failed": Server key changed or man-in-the-middle attack
  • "Too many authentication failures": Exceeded maximum authentication attempts
  • "No route to host": Network routing problem
  • "Network is unreachable": Local network configuration issue

Initial Assessment

Gathering Information

Before troubleshooting, collect essential information:

# What is the exact error message?
ssh user@server-ip 2>&1 | tee ssh-error.log

# What was working before?
# - Did this connection ever work?
# - What changed recently?
# - Can others connect?

# What is your connection method?
# - Username and authentication method (password/key)
# - Target IP address or hostname
# - Port number (if non-standard)

Quick Verification Checklist

Start with these quick checks:

  1. Can you ping the server?
  2. Is the SSH port open?
  3. Is the SSH service running (if you have console access)?
  4. Are credentials correct?
  5. Has firewall configuration changed?

Step 1: Network Connectivity Testing

Basic Reachability Tests

Verify the server is reachable on the network:

# Basic ping test
ping -c 4 server-ip

# If hostname used, test DNS resolution
nslookup server-hostname
dig server-hostname +short

# Test with both IPv4 and IPv6
ping -4 server-hostname
ping -6 server-hostname

# Continuous ping to detect intermittent issues
ping -i 0.2 server-ip

# Check network path
traceroute server-ip
mtr -c 50 server-ip

Interpreting Results:

  • No ping response: Server down, network issue, or ICMP blocked by firewall
  • High packet loss: Network congestion or unstable connection
  • High latency: Network performance issues
  • DNS resolution fails: DNS problem, not SSH issue

Port Connectivity Testing

Verify the SSH port is accessible:

# Test SSH port (default 22)
telnet server-ip 22
nc -zv server-ip 22

# If using custom port
nc -zv server-ip 2222

# Check if port is filtered
nmap -p 22 server-ip
nmap -Pn -p 22 server-ip  # Skip ping check

# Test with timeout
timeout 5 bash -c "</dev/tcp/server-ip/22" && echo "Port 22 open" || echo "Port 22 closed"

# Scan for SSH service on non-standard ports
nmap -p 1-65535 server-ip | grep ssh

Results Analysis:

  • Connection established: Port is open and reachable
  • Connection refused: Service not running or firewall blocking
  • Connection timeout: Firewall dropping packets or routing issue
  • No route to host: Network configuration problem

Step 2: SSH Client-Side Diagnostics

Verbose SSH Connection Testing

Use SSH verbose modes to see detailed connection information:

# Basic verbose mode (-v)
ssh -v user@server-ip

# More verbose (-vv)
ssh -vv user@server-ip

# Maximum verbosity (-vvv)
ssh -vvv user@server-ip 2>&1 | tee ssh-debug.log

# Verbose with specific port
ssh -vvv -p 2222 user@server-ip

# Verbose with specific identity file
ssh -vvv -i ~/.ssh/id_rsa user@server-ip

Debug Output Analysis:

Look for these key indicators in verbose output:

# Successful stages:
debug1: Connecting to server-ip [IP] port 22.
debug1: Connection established.
debug1: Remote protocol version 2.0
debug1: Server host key: ssh-rsa SHA256:xxxxx
debug1: Authentication succeeded (publickey).

# Problem indicators:
debug1: connect to address X.X.X.X port 22: Connection timed out
debug1: connect to address X.X.X.X port 22: Connection refused
debug1: Connection closed by remote host
debug1: No more authentication methods to try.

Testing Different Authentication Methods

Try various authentication approaches:

# Force password authentication
ssh -o PreferredAuthentications=password user@server-ip

# Force public key authentication
ssh -o PreferredAuthentications=publickey user@server-ip

# Disable strict host key checking (temporary, for testing)
ssh -o StrictHostKeyChecking=no user@server-ip

# Use specific identity file
ssh -i ~/.ssh/custom_key user@server-ip

# Try with different key types
ssh -o PubkeyAcceptedKeyTypes=ssh-rsa user@server-ip
ssh -o PubkeyAcceptedKeyTypes=ecdsa-sha2-nistp256 user@server-ip

# Disable all authentication methods except one
ssh -o PreferredAuthentications=publickey -o PubkeyAuthentication=yes user@server-ip

SSH Configuration File Issues

Check your local SSH client configuration:

# View your SSH config
cat ~/.ssh/config

# Check system-wide config
cat /etc/ssh/ssh_config

# Test with minimal config (ignore config files)
ssh -F /dev/null user@server-ip

# View effective configuration for host
ssh -G server-hostname

Common client config issues:

# Check for problematic directives
grep -i "Host\|HostName\|Port\|IdentityFile" ~/.ssh/config

# Verify file permissions
ls -la ~/.ssh/config
# Should be: -rw------- (600)

# Fix permissions if needed
chmod 600 ~/.ssh/config

Step 3: SSH Key Diagnostics

Verifying SSH Keys

Check your SSH key setup:

# List your SSH keys
ls -la ~/.ssh/

# View public key
cat ~/.ssh/id_rsa.pub
cat ~/.ssh/id_ed25519.pub

# Check key permissions
ls -l ~/.ssh/id_rsa
# Should be: -rw------- (600)

# Fix key permissions
chmod 700 ~/.ssh
chmod 600 ~/.ssh/id_rsa
chmod 644 ~/.ssh/id_rsa.pub

# Test key validity
ssh-keygen -l -f ~/.ssh/id_rsa
ssh-keygen -y -f ~/.ssh/id_rsa

# Check if key is loaded in ssh-agent
ssh-add -l

# Add key to ssh-agent
eval $(ssh-agent)
ssh-add ~/.ssh/id_rsa

Testing Key-Based Authentication

Verify key-based authentication setup:

# Copy key to server (if you have password access)
ssh-copy-id user@server-ip
ssh-copy-id -i ~/.ssh/id_rsa.pub user@server-ip

# Manual key verification (what ssh-copy-id does)
cat ~/.ssh/id_rsa.pub | ssh user@server-ip "mkdir -p ~/.ssh && chmod 700 ~/.ssh && cat >> ~/.ssh/authorized_keys && chmod 600 ~/.ssh/authorized_keys"

# Test specific key
ssh -i ~/.ssh/id_rsa -vvv user@server-ip

# Verify key format
ssh-keygen -l -f ~/.ssh/id_rsa.pub

Common Key Problems

# Check for multiple keys causing "too many authentication failures"
ssh-add -l
# If too many keys, clear and add specific key
ssh-add -D
ssh-add ~/.ssh/id_rsa

# Generate new key pair if corrupted
ssh-keygen -t ed25519 -C "[email protected]"
ssh-keygen -t rsa -b 4096 -C "[email protected]"

# Convert old key format to new
ssh-keygen -p -f ~/.ssh/id_rsa -m pem

Step 4: Server-Side Diagnostics

Accessing Server Console

If SSH is unavailable, use alternative access:

  1. Cloud Provider Console: AWS EC2 Console, DigitalOcean Console, etc.
  2. IPMI/iLO/iDRAC: Out-of-band management
  3. VNC/KVM: Remote desktop access
  4. Physical Access: Direct keyboard/monitor

Checking SSH Service Status

Once you have console access:

# Check SSH service status
systemctl status sshd
systemctl status ssh

# Check if SSH is running
ps aux | grep sshd
pgrep sshd

# Restart SSH service
systemctl restart sshd
systemctl restart ssh

# Enable SSH on boot
systemctl enable sshd

# Check SSH service startup errors
journalctl -u sshd -n 50
systemctl status sshd -l

SSH Service Port Verification

Verify which port SSH is listening on:

# Check listening ports
ss -tlnp | grep sshd
netstat -tlnp | grep sshd

# Verify SSH configuration port
grep "^Port" /etc/ssh/sshd_config

# Check all SSH listening addresses
lsof -i -P | grep sshd

SSH Configuration File Analysis

Check server SSH daemon configuration:

# View SSH daemon config
cat /etc/ssh/sshd_config

# Test configuration validity
sshd -t
sshd -T  # Show effective configuration

# Check for common issues
grep -E "^PermitRootLogin|^PasswordAuthentication|^PubkeyAuthentication|^Port|^ListenAddress" /etc/ssh/sshd_config

# Backup and view config without comments
grep -v "^#" /etc/ssh/sshd_config | grep -v "^$"

Critical Configuration Directives:

# Allow root login?
grep "PermitRootLogin" /etc/ssh/sshd_config
# Should be: PermitRootLogin no (best practice)

# Allow password authentication?
grep "PasswordAuthentication" /etc/ssh/sshd_config
# PasswordAuthentication yes (or no if key-only)

# Allow public key authentication?
grep "PubkeyAuthentication" /etc/ssh/sshd_config
# PubkeyAuthentication yes

# Which users can connect?
grep "AllowUsers\|DenyUsers\|AllowGroups" /etc/ssh/sshd_config

# Maximum authentication attempts
grep "MaxAuthTries" /etc/ssh/sshd_config

Checking Authorized Keys

Verify authorized_keys file on server:

# Check authorized_keys file
cat ~/.ssh/authorized_keys

# Check file permissions
ls -la ~/.ssh/
ls -l ~/.ssh/authorized_keys

# Correct permissions
chmod 700 ~/.ssh
chmod 600 ~/.ssh/authorized_keys

# Check ownership
ls -l ~/.ssh/authorized_keys
# Should be owned by the user

# Fix ownership
chown user:user ~/.ssh/authorized_keys
chown -R user:user ~/.ssh/

# Verify key format in authorized_keys
grep "ssh-rsa\|ssh-ed25519\|ecdsa" ~/.ssh/authorized_keys

# Check for invalid entries
awk '{print NF, $1}' ~/.ssh/authorized_keys

Step 5: Firewall and Security Diagnostics

Checking Firewall Rules

Firewalls are common causes of SSH connectivity issues:

# Check iptables rules
iptables -L -n -v
iptables -L INPUT -n -v | grep -E "22|ssh"

# Check if SSH port is allowed
iptables -L INPUT -n | grep "dpt:22"

# Temporarily disable firewall (for testing only!)
systemctl stop iptables
systemctl stop firewalld
ufw disable

# Check UFW status
ufw status verbose
ufw status numbered

# Check firewalld
firewall-cmd --list-all
firewall-cmd --list-ports
firewall-cmd --list-services | grep ssh

# Check nftables
nft list ruleset | grep -i ssh

Allowing SSH Through Firewall

# UFW
ufw allow 22/tcp
ufw allow ssh
ufw allow from specific-ip to any port 22

# firewalld
firewall-cmd --permanent --add-service=ssh
firewall-cmd --permanent --add-port=22/tcp
firewall-cmd --reload

# iptables
iptables -A INPUT -p tcp --dport 22 -j ACCEPT
iptables-save > /etc/iptables/rules.v4

Checking TCP Wrappers

TCP Wrappers can block SSH connections:

# Check hosts.allow
cat /etc/hosts.allow
grep sshd /etc/hosts.allow

# Check hosts.deny
cat /etc/hosts.deny
grep sshd /etc/hosts.deny

# Allow all SSH (for testing)
echo "sshd: ALL" >> /etc/hosts.allow

# Allow specific IP
echo "sshd: 192.168.1.100" >> /etc/hosts.allow

Checking Fail2Ban

Fail2Ban may have banned your IP:

# Check if fail2ban is running
systemctl status fail2ban

# Check banned IPs
fail2ban-client status sshd
iptables -L -n | grep DROP

# Unban your IP
fail2ban-client set sshd unbanip YOUR-IP

# Check fail2ban log
tail -100 /var/log/fail2ban.log
grep "Ban" /var/log/fail2ban.log

SELinux Issues

SELinux can interfere with SSH:

# Check SELinux status
getenforce
sestatus

# Check SELinux denials
ausearch -m avc -ts recent
grep "sshd" /var/log/audit/audit.log

# Temporarily disable (for testing)
setenforce 0

# Check SSH SELinux context
ls -Z /etc/ssh/sshd_config
ls -Z ~/.ssh/authorized_keys

# Restore correct context
restorecon -Rv /etc/ssh/
restorecon -Rv ~/.ssh/

Step 6: Authentication Issues

Password Authentication Problems

# Test password authentication explicitly
ssh -o PreferredAuthentications=password user@server-ip

# Check if password authentication is enabled
grep "PasswordAuthentication" /etc/ssh/sshd_config

# Check PAM configuration
cat /etc/pam.d/sshd

# Check account status
passwd -S username
chage -l username

# Unlock account if locked
passwd -u username

Key Authentication Problems

# Check authorized_keys file exists and has content
[ -f ~/.ssh/authorized_keys ] && echo "File exists" || echo "File missing"
wc -l ~/.ssh/authorized_keys

# Verify key in authorized_keys matches your public key
# On client:
cat ~/.ssh/id_rsa.pub
# On server:
grep "$(cat ~/.ssh/id_rsa.pub | cut -d' ' -f2)" ~/.ssh/authorized_keys

# Check StrictModes
grep "StrictModes" /etc/ssh/sshd_config
# If yes, home directory must not be writable by others
ls -ld /home/username
chmod 755 /home/username

Permission Problems

SSH is strict about file permissions:

# Check all SSH-related permissions
# Home directory
ls -ld /home/username  # Should be 755 or 700

# .ssh directory
ls -ld /home/username/.ssh  # Should be 700

# authorized_keys
ls -l /home/username/.ssh/authorized_keys  # Should be 600

# Fix all permissions
chmod 755 /home/username
chmod 700 /home/username/.ssh
chmod 600 /home/username/.ssh/authorized_keys
chown -R username:username /home/username/.ssh

Step 7: Log Analysis

SSH Server Logs

Check SSH daemon logs for connection attempts:

# Recent SSH log entries
journalctl -u sshd -n 100
journalctl -u ssh -n 100

# Follow SSH logs in real-time
journalctl -u sshd -f

# Authentication logs
tail -100 /var/log/auth.log  # Debian/Ubuntu
tail -100 /var/log/secure    # CentOS/RHEL

# Search for specific user
grep "user@" /var/log/auth.log
journalctl -u sshd | grep "username"

# Failed authentication attempts
grep "Failed password" /var/log/auth.log | tail -20
grep "Connection closed" /var/log/auth.log | tail -20

# Successful logins
grep "Accepted" /var/log/auth.log | tail -20

# Key-related errors
grep "publickey" /var/log/auth.log | tail -20

Common Log Error Messages

# "Permission denied (publickey)"
# Cause: Key authentication failed
# Check: authorized_keys, key permissions

# "User username from IP not allowed"
# Cause: AllowUsers/DenyUsers restriction
# Check: /etc/ssh/sshd_config

# "Authentication refused: bad ownership or modes"
# Cause: Wrong permissions on .ssh or authorized_keys
# Fix: chmod 700 ~/.ssh && chmod 600 ~/.ssh/authorized_keys

# "Could not load host key"
# Cause: Missing or corrupted host keys
# Fix: ssh-keygen -A

# "Connection closed by authenticating user"
# Cause: Various authentication failures
# Check: journalctl -u sshd for details

Debug Mode SSH Daemon

Run SSH daemon in debug mode (careful - stops normal SSH):

# Stop SSH service
systemctl stop sshd

# Run in debug mode
/usr/sbin/sshd -d -p 2222

# In another terminal, connect
ssh -p 2222 user@localhost

# For more verbose output
/usr/sbin/sshd -ddd -p 2222

# When done, stop debug daemon and restart normal service
systemctl start sshd

Root Cause Analysis

Common SSH Connectivity Issues

1. Network/Firewall Issues

  • Firewall blocking port 22
  • Wrong IP address or hostname
  • Network routing problems
  • Cloud provider security groups

2. SSH Service Issues

  • SSH daemon not running
  • Wrong port configuration
  • Service crashed or failing to start
  • Configuration syntax errors

3. Authentication Issues

  • Wrong username or password
  • SSH keys not properly configured
  • authorized_keys file problems
  • Permission issues on files/directories

4. Configuration Issues

  • PermitRootLogin set to no
  • PasswordAuthentication disabled
  • User not in AllowUsers list
  • MaxAuthTries exceeded

5. System Issues

  • Disk full preventing writes
  • Too many open files
  • SELinux/AppArmor blocking
  • PAM configuration problems

Solutions and Remediation

Quick Fixes for Common Problems

Connection Timeout:

# Check and allow SSH through firewall
ufw allow 22/tcp
firewall-cmd --add-service=ssh --permanent && firewall-cmd --reload

# Verify SSH is running
systemctl start sshd && systemctl enable sshd

Connection Refused:

# Start SSH service
systemctl start sshd

# Check SSH is listening
ss -tlnp | grep :22

Permission Denied:

# Fix SSH file permissions
chmod 700 ~/.ssh
chmod 600 ~/.ssh/authorized_keys
chmod 600 ~/.ssh/id_rsa
chmod 644 ~/.ssh/id_rsa.pub
chown -R $USER:$USER ~/.ssh

Host Key Verification Failed:

# Remove old key from known_hosts
ssh-keygen -R server-hostname
ssh-keygen -R server-ip

# Regenerate server host keys (on server)
rm /etc/ssh/ssh_host_*
ssh-keygen -A
systemctl restart sshd

Too Many Authentication Failures:

# Clear ssh-agent and add only needed key
ssh-add -D
ssh-add ~/.ssh/id_rsa

# Or specify key explicitly
ssh -i ~/.ssh/specific_key user@server

Emergency Access Recovery

If completely locked out:

  1. Use cloud provider console (AWS, DigitalOcean, etc.)
  2. Use IPMI/iLO/iDRAC for bare metal
  3. Boot from rescue mode and fix configuration
  4. Contact hosting provider for console access

Restoring SSH Configuration

# Backup current config
cp /etc/ssh/sshd_config /etc/ssh/sshd_config.backup

# Restore default configuration
# Debian/Ubuntu
cp /usr/share/openssh/sshd_config /etc/ssh/sshd_config

# Or create minimal working config
cat > /etc/ssh/sshd_config << 'EOF'
Port 22
PermitRootLogin no
PubkeyAuthentication yes
PasswordAuthentication yes
ChallengeResponseAuthentication no
UsePAM yes
X11Forwarding yes
PrintMotd no
AcceptEnv LANG LC_*
Subsystem sftp /usr/lib/openssh/sftp-server
EOF

# Test and restart
sshd -t && systemctl restart sshd

Prevention and Best Practices

Secure SSH Configuration

# Recommended /etc/ssh/sshd_config settings
cat >> /etc/ssh/sshd_config << 'EOF'
# Security hardening
PermitRootLogin no
PasswordAuthentication no
PubkeyAuthentication yes
ChallengeResponseAuthentication no
UsePAM yes

# Limit users
AllowUsers username1 username2

# Connection settings
MaxAuthTries 3
LoginGraceTime 60
ClientAliveInterval 300
ClientAliveCountMax 2

# Disable unused features
X11Forwarding no
PermitTunnel no
AllowAgentForwarding no
EOF

sshd -t && systemctl restart sshd

Monitoring and Alerting

# Create SSH monitoring script
cat > /usr/local/bin/ssh-monitor.sh << 'EOF'
#!/bin/bash
if ! systemctl is-active --quiet sshd; then
    echo "SSH service is down!" | mail -s "SSH Alert" [email protected]
    systemctl start sshd
fi

# Check for excessive failed attempts
FAILED=$(grep "Failed password" /var/log/auth.log | tail -10 | wc -l)
if [ $FAILED -gt 5 ]; then
    echo "High number of failed SSH attempts" | mail -s "SSH Security Alert" [email protected]
fi
EOF

chmod +x /usr/local/bin/ssh-monitor.sh

# Add to crontab
echo "*/5 * * * * /usr/local/bin/ssh-monitor.sh" | crontab -

Backup Access Methods

Always maintain backup access:

# Configure multiple SSH ports
cat >> /etc/ssh/sshd_config << 'EOF'
Port 22
Port 2222
EOF

# Allow both in firewall
ufw allow 22/tcp
ufw allow 2222/tcp

# Test configuration and restart
sshd -t && systemctl restart sshd

SSH Bastion/Jump Host

Set up jump host for added security:

# On client, configure SSH config
cat >> ~/.ssh/config << 'EOF'
Host bastion
    HostName bastion-ip
    User username
    IdentityFile ~/.ssh/bastion_key

Host internal-server
    HostName internal-ip
    User username
    IdentityFile ~/.ssh/internal_key
    ProxyJump bastion
EOF

# Connect through bastion
ssh internal-server

Regular Maintenance

# Weekly SSH health check script
cat > /usr/local/bin/ssh-health-check.sh << 'EOF'
#!/bin/bash
echo "SSH Health Check - $(date)" > /tmp/ssh-health.txt

# Check SSH service
systemctl is-active sshd >> /tmp/ssh-health.txt

# Check listening ports
ss -tlnp | grep sshd >> /tmp/ssh-health.txt

# Check recent failed attempts
echo "Failed attempts in last 24h:" >> /tmp/ssh-health.txt
grep "Failed password" /var/log/auth.log | grep "$(date +%b\ %d)" | wc -l >> /tmp/ssh-health.txt

# Check file permissions
echo "SSH directory permissions:" >> /tmp/ssh-health.txt
ls -ld ~/.ssh >> /tmp/ssh-health.txt
ls -l ~/.ssh/authorized_keys >> /tmp/ssh-health.txt

mail -s "SSH Health Report" [email protected] < /tmp/ssh-health.txt
EOF

chmod +x /usr/local/bin/ssh-health-check.sh

Advanced Troubleshooting

SSH Tunneling and Port Forwarding Issues

# Local port forwarding
ssh -L 8080:localhost:80 user@server -v

# Remote port forwarding
ssh -R 8080:localhost:80 user@server -v

# Dynamic SOCKS proxy
ssh -D 1080 user@server -v

# Check if forwarding is allowed
grep "AllowTcpForwarding\|GatewayPorts" /etc/ssh/sshd_config

X11 Forwarding Problems

# Enable X11 forwarding
ssh -X user@server

# Check X11 configuration
grep "X11Forwarding" /etc/ssh/sshd_config
grep "X11UseLocalhost" /etc/ssh/sshd_config

# Test X11
echo $DISPLAY
xauth list

MTU and Network Issues

# Test MTU size
ping -M do -s 1472 server-ip

# Adjust MTU if needed
ip link set dev eth0 mtu 1400

# SSH with MTU consideration
ssh -o "IPQoS=throughput" user@server

Conclusion

SSH connectivity issues can stem from numerous sources including network problems, firewall configurations, authentication failures, or service issues. The key to effective troubleshooting is a systematic approach:

  1. Start with basics: Network connectivity and service status
  2. Use verbose mode: SSH -vvv provides invaluable debugging information
  3. Check logs: Server logs reveal most authentication and configuration issues
  4. Verify permissions: SSH is strict about file permissions
  5. Test incrementally: Isolate variables by testing one thing at a time
  6. Document changes: Keep track of what you've modified
  7. Maintain backup access: Always have alternative access methods

Regular maintenance, proper monitoring, secure configuration, and understanding of SSH internals will help prevent most connectivity issues. When problems do occur, this systematic diagnostic approach will help you quickly identify and resolve them.

Remember to always test changes in non-production environments first, maintain backups of configuration files, and document your troubleshooting steps for future reference.