Skip to content

Lab: SSH Troubleshooting

Pairs with: SSH Hardening and Key-Based Authentication

Environment: A single disposable VM or container with openssh-server installed, and a second machine (or your host) to connect from. Do not run this against any host you can't afford to be temporarily locked out of.

Starting conditions

  1. A fresh VM with OpenSSH server installed and running (systemctl status sshd / systemctl status ssh shows active (running)).
  2. A non-root user account you can log into with a password, for setup purposes.
  3. Console/hypervisor access to the VM independent of SSH (critical — several steps below can lock out SSH access, and you need a way back in that doesn't depend on SSH itself).

Instructions

Part 1 — Break it, deliberately

  1. Generate an SSH key pair on your client machine and deploy the public key to the VM.
  2. Confirm key-based login works.
  3. Introduce one of the following faults (pick one, note which you picked):
  4. Wrong permissions: chmod 644 ~/.ssh/authorized_keys on the VM — wait, this is already correct; instead run chmod 777 ~/.ssh (the directory itself) on the VM.
  5. Wrong sshd_config setting: set PubkeyAuthentication no and restart sshd.
  6. Firewall block: block port 22 from your client's IP specifically (not globally — you need another way to fix it, which is part of the diagnosis).
  7. Attempt to connect. It should fail.

Part 2 — Diagnose it without being told the answer

Use ssh -v (or -vvv for maximum detail) from the client, and the server's own logs (journalctl -u sshd or /var/log/auth.log depending on distribution), to determine:

  1. Does the client even reach the server at the TCP level, or does the connection fail before authentication is attempted at all?
  2. If it reaches authentication, which authentication method is being attempted, and how does the server respond to it?
  3. What, specifically, in the server's own log corresponds to the failure?

Expected results

  • With the wrong .ssh permissions fault: ssh -v will show the client successfully connecting and offering the key, but authentication fails; the server's log (not the client's) will show the actual reason — typically something like Authentication refused: bad ownership or modes for directory /home/user/.ssh.
  • With PubkeyAuthentication no: ssh -v shows the server's advertised supported auth methods excluding publickey entirely — visible client-side, no server log inspection needed to spot this one.
  • With the firewall block: the connection hangs and eventually times out at the TCP level, before any SSH protocol banner is exchanged — ssh -v shows nothing past "Connecting to \<host>", confirming this is a network-layer problem, not an SSH-layer one.

Diagnosis checklist

  • [ ] Did you check the server's log, not just the client's -v output? (Required for the permissions fault — the client alone doesn't show why the key was rejected.)
  • [ ] Did you distinguish "connection refused/timed out" (network layer) from "authentication failed" (SSH layer, connection succeeded)?
  • [ ] For the permissions fault specifically: what are the correct permissions for ~/.ssh and ~/.ssh/authorized_keys, and how do you confirm them with stat?

Success criteria

You've completed this lab when you can, from a cold start, intentionally introduce any one of these three faults in a colleague's VM, and correctly identify — using only ssh -v and the server's logs — which of the three it was, without being told in advance.

Cleanup / rollback

# Restore .ssh permissions if you used that fault:
chmod 700 ~/.ssh
chmod 600 ~/.ssh/authorized_keys

# Restore sshd_config if you changed it:
sudo sed -i 's/^PubkeyAuthentication no/PubkeyAuthentication yes/' /etc/ssh/sshd_config
sudo sshd -t && sudo systemctl restart sshd

# Remove any firewall rule you added for the test

Confirm ssh -v succeeds with Authentication succeeded (publickey) again before considering the lab environment restored.