How to Connect to a Linux Server via SSH with Keys

SSH (Secure Shell) is the standard way to open an encrypted terminal session on a remote Linux server. In this tutorial you will connect to a server for the first time with a password, verify its host key, switch to key-based authentication with an Ed25519 key pair, and save the connection in an SSH config file so you can log in with a short alias. The client steps work on Linux, macOS and Windows 10/11; the server can be any Linux distribution running OpenSSH, such as Ubuntu 24.04.

Prerequisites

To follow this guide you need:

A Linux server with the OpenSSH server running, for example a CubePath VPS with Ubuntu 24.04.
The server's public IP address (shown as your_server_ip in this guide), the login user (often root on a new server) and its password or an existing key.
A local computer running Linux, macOS, or Windows 10/11.
TCP port 22 reachable from your network (or the custom port your server uses).

Step 1 - Checking the SSH client on your computer

All modern desktop systems ship an OpenSSH client, so there is usually nothing to install. Open a terminal (on Windows, open PowerShell or Windows Terminal) and check the version:

ssh -V

OpenSSH_9.6p1 Ubuntu-3ubuntu13.5, OpenSSL 3.0.13 30 Jan 2024

Any OpenSSH 8.x or newer is fine. If the command is not found:

Debian/Ubuntu desktop: sudo apt install openssh-client
Fedora/Rocky/RHEL: sudo dnf install openssh-clients
Windows: open Settings > System > Optional features, choose Add a feature and install OpenSSH Client, then open a new PowerShell window.

The rest of this tutorial uses the same ssh, ssh-keygen and scp commands on every platform. The only difference on Windows is how the public key is copied in Step 4.

Step 2 - Connecting for the first time with a password

The basic syntax is ssh user@host. Replace your_server_ip with the address of your server:

ssh root@your_server_ip

If the server listens on a non-standard port, add -p:

ssh -p 2222 root@your_server_ip

On the first connection the client does not know the server yet and shows its host key fingerprint:

The authenticity of host '203.0.113.10 (203.0.113.10)' can't be established.
ED25519 key fingerprint is SHA256:Qm3uCz2XlY0wq9mH6bJ0V7yE0mZt3Q9f1s0bT2yN8aE.
This key is not known by any other names.
Are you sure you want to continue connecting (yes/no/[fingerprint])?

This prompt protects you against someone intercepting the connection. To be certain you are talking to your server, compare the fingerprint with the one the server itself reports. Open the web console of your server (on CubePath, the VNC console of the VPS in the dashboard), log in, and run:

ssh-keygen -lf /etc/ssh/ssh_host_ed25519_key.pub

256 SHA256:Qm3uCz2XlY0wq9mH6bJ0V7yE0mZt3Q9f1s0bT2yN8aE root@server (ED25519)

If both fingerprints match, type yes and press ENTER. The client stores the key in ~/.ssh/known_hosts and will warn you loudly if it ever changes. Then type the password; nothing is echoed while you type.

A successful login ends at a shell prompt on the server:

Welcome to Ubuntu 24.04.3 LTS (GNU/Linux 6.8.0-79-generic x86_64)
...
root@server:~#

Type exit or press CTRL+D to close the session.

Step 3 - Generating an SSH key pair

Passwords can be guessed and are sent to the server on every login. A key pair is safer: the private key never leaves your computer, and the server only stores the public half. Ed25519 is the recommended key type today; it is short, fast and supported by every current OpenSSH server.

Run this on your local computer, not on the server:

ssh-keygen -t ed25519 -C "[email protected]"

Press ENTER to accept the default location and then type a passphrase. The passphrase encrypts the private key on disk, so a stolen laptop does not mean a stolen server.

Generating public/private ed25519 key pair.
Enter file in which to save the key (/home/sammy/.ssh/id_ed25519):
Enter passphrase for "/home/sammy/.ssh/id_ed25519" (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/sammy/.ssh/id_ed25519
Your public key has been saved in /home/sammy/.ssh/id_ed25519.pub

This creates two files:

File	Contents	Share it?
`~/.ssh/id_ed25519`	Private key	Never
`~/.ssh/id_ed25519.pub`	Public key	Yes, this goes on the server

On Windows the same files are created in C:\Users\your_user\.ssh\.

Note

If you must connect to a very old system that does not support Ed25519, create an RSA key instead with ssh-keygen -t rsa -b 4096.

Step 4 - Copying the public key to the server

The server accepts a key when its public half is listed in ~/.ssh/authorized_keys of the user you log in as.

On Linux and macOS

ssh-copy-id connects with your password one last time and appends the key with the correct permissions:

ssh-copy-id root@your_server_ip

For a custom port use ssh-copy-id -p 2222 root@your_server_ip.

/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
[email protected]'s password:

Number of key(s) added: 1

Now try logging into the machine, with:   "ssh '[email protected]'"
and check to make sure that only the key(s) you wanted were added.

On Windows (PowerShell)

Windows does not include ssh-copy-id. This one-liner reads your public key and appends it on the server, creating the directory with the right permissions if needed:

type $env:USERPROFILE\.ssh\id_ed25519.pub | ssh root@your_server_ip "mkdir -p ~/.ssh && chmod 700 ~/.ssh && cat >> ~/.ssh/authorized_keys && chmod 600 ~/.ssh/authorized_keys"

Testing the key

Connect again:

ssh root@your_server_ip

This time you are asked for the key's passphrase (a local prompt) instead of the server password:

Enter passphrase for key '/home/sammy/.ssh/id_ed25519':

If you still get a password: prompt, the key was not accepted. See the Troubleshooting section before continuing.

Step 5 - Caching the passphrase with ssh-agent

Typing the passphrase for every connection gets tedious. ssh-agent keeps the decrypted key in memory for your session.

On Linux, most desktop sessions already run an agent. Add the key with:

ssh-add ~/.ssh/id_ed25519

On macOS, store the passphrase in the Keychain so it survives reboots:

ssh-add --apple-use-keychain ~/.ssh/id_ed25519

On Windows, the agent is a service that is disabled by default. In an administrator PowerShell:

Get-Service ssh-agent | Set-Service -StartupType Automatic
Start-Service ssh-agent

Then, in a normal PowerShell window:

ssh-add $env:USERPROFILE\.ssh\id_ed25519

Check which keys the agent holds:

ssh-add -l

256 SHA256:3Jd0N1m8gq7sK7c0Qv9eZ2rVb4P8XyWmT5uLkHcA1sE [email protected] (ED25519)

Step 6 - Saving connections in ~/.ssh/config

Instead of remembering IP addresses, users and ports, describe each server once in the client config file. Create or open it on your local computer:

nano ~/.ssh/config

Add a block per server:

Host web1
    HostName your_server_ip
    User root
    Port 22
    IdentityFile ~/.ssh/id_ed25519
    IdentitiesOnly yes
    ServerAliveInterval 60

What each line does:

Host web1: the alias you will type.
HostName, User, Port: the real connection details.
IdentityFile and IdentitiesOnly yes: offer only this key, which avoids "Too many authentication failures" when the agent holds many keys.
ServerAliveInterval 60: sends a keepalive every 60 seconds so idle sessions are not dropped by NAT routers.

On Linux and macOS, make sure only you can read the file:

chmod 600 ~/.ssh/config

Now connect with the alias:

ssh web1

The alias works with the other OpenSSH tools too, for example to copy a file to the server's /tmp directory:

scp ./backup.tar.gz web1:/tmp/

Step 7 - Disabling password logins (optional)

Once key login works, you can stop the server from accepting passwords at all, which ends password brute-force attempts. Do this only after confirming in Step 4 that your key works, and keep your current session open while you test.

On the server, create a drop-in file. OpenSSH uses the first value it reads for each option, and files in /etc/ssh/sshd_config.d/ are read in alphabetical order before the main file, so a name starting with 00- takes precedence over files such as 50-cloud-init.conf:

sudo nano /etc/ssh/sshd_config.d/00-keys-only.conf

PasswordAuthentication no
KbdInteractiveAuthentication no

Check the syntax and the effective values, then restart the service (ssh on Ubuntu and Debian, sshd on Rocky Linux and other RHEL-based systems):

sudo sshd -t
sudo sshd -T | grep -Ei '^(passwordauthentication|kbdinteractiveauthentication)'
sudo systemctl restart ssh

passwordauthentication no
kbdinteractiveauthentication no

From a new local terminal, confirm that the server now refuses passwords:

ssh -o PubkeyAuthentication=no root@your_server_ip

[email protected]: Permission denied (publickey).

A normal ssh root@your_server_ip with your key should still work.

Troubleshooting

Connection timed out: the packets never reach SSH. Check the IP address, that the server is running, and that a firewall (UFW, firewalld, or a network firewall in front of the server) allows the SSH port.

Connection refused: the server is reachable but nothing listens on that port. You may be using the wrong port, or the SSH service is stopped. From the web console, run sudo systemctl status ssh (or sshd on RHEL-based systems) and sudo ss -tlnp | grep ssh.

Permission denied (publickey): the server did not accept your key. Run the client in verbose mode to see which keys it offers:

ssh -v root@your_server_ip

On the server, the usual causes are wrong ownership or permissions. Fix them for the target user:

chmod 700 ~/.ssh
chmod 600 ~/.ssh/authorized_keys

The server-side reason is logged in the journal: sudo journalctl -u ssh -n 50 (-u sshd on RHEL-based systems).

WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!: the server presents a different host key than the one you saved. This is expected after reinstalling the server with the same IP. If you know that is the case, remove the old entry and connect again, verifying the new fingerprint as in Step 2:

ssh-keygen -R your_server_ip

If you did not reinstall anything, do not connect: investigate first.

Conclusion

You can now connect to your server over SSH, verify its identity, log in with an Ed25519 key protected by a passphrase, and reach it with a short alias. Good next steps are creating a non-root sudo user and hardening the server (firewall, Fail2Ban, automatic updates), changing the default SSH port if you want less log noise, and adding a separate key for each computer you work from so you can revoke one without affecting the others.

On this page