OpenSSH/Logging and Troubleshooting

Both the OpenSSH client and server offer a lot of choice as to where the logs are written and how much information is collected. A prerequisite for logging is having an accurate system clock using the Network Time Protocol, NTP, or equivalent, service which provides ongoing time synchronization with rest of the world. The more accurate the time stamp in the log is, the faster it is to coordinate forensics between machines or sites or service providers. If you have to contact outside parties like a service provider, progress can usually only be made with very exact times. Server Logs By default sshd(8) sends logging information to the system logs using the log level INFO and the system log facility AUTH. So the place to look for log data from sshd(8) is in /var/log/auth.log. These defaults can be overridden using the SyslogFacility and LogLevel directives. Below is a typical server startup entry in the authorization log. Mar 19 14:45:40 server sshd[21157]: Server listening on 0.0.0.0 port 22. Mar 19 14:45:40 server sshd[21157]: Server listening on :: port 22. In most cases the default level of logging is sufficient, but during initial testing of new services or activities it is sometimes necessary to have more information. Debugging info usually goes to stderr. Starting with OpenSSH 7.6, Match blocks can set alternate log levels for specific conditions. The log excerpt below show the same basic server start up with increased detail. Contrast the log level DEBUG1 below with the default above: Apr 5 12:39:14 server sshd[85029]: debug1: Bind to port 22 on 0.0.0.0. Apr 5 12:39:14 server sshd[85029]: Server listening on 0.0.0.0 port 22. Apr 5 12:39:14 server sshd[85029]: debug1: Bind to port 22 on ::. Apr 5 12:39:14 server sshd[85029]: Server listening on :: port 22. And here is the same startup using the most verbose level, DEBUG3: Apr 5 12:40:59 server sshd[49707]: debug2: fd 6 setting O_NONBLOCK Apr 5 12:40:59 server sshd[49707]: debug1: Bind to port 22 on 0.0.0.0. Apr 5 12:40:59 server sshd[49707]: Server listening on 0.0.0.0 port 22. Apr 5 12:40:59 server sshd[49707]: debug2: fd 7 setting O_NONBLOCK Apr 5 12:40:59 server sshd[49707]: debug1: Bind to port 22 on ::. Apr 5 12:40:59 server sshd[49707]: Server listening on :: port 22. Every failed login attempt is recorded, once the value in directive MaxAuthTries is exceeded the connection is broken. Below is a log excerpt showing how the default log looks after some failed attempts: ... Apr 5 12:44:43 server sshd-session[80932]: Failed password for root from 103.42.56.121 port 58164 ssh2 Apr 5 12:44:51 server last message repeated 2 times Apr 5 12:44:51 server sshd-session[80932]: Connection closed by authenticating user root 103.42.56.121 port 58164 [preauth] ... It is not usually a good idea to allow root login, especially not root login with authentication via password. Blocking login for root simplifies the log analysis greatly, and in particular it eliminates the time consuming question of who is trying to get in and why. People that need full root level access can gain it through su(1) for general activities. Or for specific tasks which need root level access, those can be given those privileges through custom-made entries for sudo(8) or doas(1). Note that in those cases, only specific services and programs should be allowed, not blanket access which is an all too common misconfiguration seen with sudo(8). Alternatively, a single-purpose key made using forced-commands-only could be used since some argue that providing extra means of privilege escalation, such as su(1), sudo(8), or doas(1), is more dangerous than carefully providing remote root access through a key or certificate tied to a specific function. Successful logins By default, the server does not store much information about user transactions. That is a good thing. It is also a good thing to recognize when the system is operating as it should. So here is an example of a successful SSH login: Apr 4 13:30:08 server sshd-session[86956]: Accepted password for fred from 198.51.100.135 port 52646 ssh2 And here is an example using a key for authentication. It shows the key fingerprint as a SHA256 hash in base64. Apr 5 12:49:11 server sshd-session[41210]: Accepted publickey for fred from 198.51.100.135 port 28927 ssh2: ED25519 SHA256:03u96bPG5qYi7hIEohNDfxOW+KOpMS9LPi4YdiCtGaI And here is an example of successful authentication with a user certificate. The certificate's identification string is "foobar" and the serial number is "9624". In this example the certificate is using ECDSA and the key itself is using Ed25519. The certificate, being a different key of its own, has a different SHA256 fingerprint from the authentication key itself. Apr 5 13:50:57 server sshd-session[20263]: Accepted publickey for fred from 127.0.0.1 port 41783 ssh2: ED25519-CERT SHA256:UP1+mGQ5tfT84YCRVgytAKtikn0ujmfhXKdoxRoWDtU ID edcba (serial 2) CA ED25519 SHA256:4lngxTgobhprXfPY/p+2HNMBg6PxNDaOWrvei6kScrM In all three cases, older versions prior to 9.8 will use a process named sshd rather than one named sshd-session as is now a side effect of better privilege separation. Prior to 6.8, the key's fingerprint was a hexadecimal MD5 hash. Jan 28 11:51:43 server sshd[5104]: Accepted publickey for fred from 192.0.2.60 port 60594 ssh2: RSA e8:31:68:c7:01:2d:25:20:36:8f:50:5d:f9:ee:70:4c In older versions of OpenSSH prior to 6.3 the key fingerprint is completely missing from authentication logging. Jan 28 11:52:05 server sshd[1003]: Accepted publickey for fred from 192.0.2.60 port 20042 ssh2 Here is an example of password authentication for an SFTP session, using the server's internal-sftp subsystem. The logging for that subsystem set to INFO. Mar 14 20:14:18 server sshd-session[19850]: Accepted password for fred from 192.0.2.60 port 59946 ssh2 Here is an example of a successful SFTP login using an Ed25519 key for authentication, with the SFTP subsystem itself logging at INFO. Apr 5 13:56:02 server shd-session[58881]: Accepted publickey for fred from 192.0.2.60 port 40445 ssh2: ED25519 SHA256:UP1+mGQ5tfT84YCRVgytAKtikn0ujmfhXKdoxRoWDtU Apr 5 13:56:02 server internal-sftp[21121]: session opened for local user fred from [192.0.2.60] Additional data, such as connection duration, can be logged with the help of xinetd(8). Logging Problems from SSH Certificate Authentication When a certificate fails, there is generally no client side information disclosed and all investigation must occur server side. Finding the account or actual certificate in question can require some sleuthing in older versions of OpenSSH, but newer versions can identify the certificate and the certificate authority under certain circumstances. It is also important to be familiar with how log entries are supposed to look when things go correctly. Here is a successful login using certificate-based authentication: Aug 7 18:32:29 server sshd-session[49021]: Accepted publickey for fred from 192.0.2.60 port 10932 ssh2: ED25519-CERT SHA256:UP1+mGQ5tfT84YCRVgytAKtikn0ujmfhXKdoxRoWDtU ID fedcba (serial 11) CA ED25519 SHA256:4lngxTgobhprXfPY/p+2HNMBg6PxNDaOWrvei6kScrM When the connection is closed there will be a log entry noting which account was involved. Older versions of OpenSSH will have less information in the log. May 5 16:02:50 server sshd-session[32110]: Received disconnect from 192.0.2.60 port 12952:11: disconnected by user May 5 16:02:50 server sshd-session[32110]: Disconnected from user fred 192.0.2.60 port 12952 However, if the connection is allowed to timeout without first making any other authentication attempts by some other means, then there will be nothing to go on except maybe the time of day. Apr 5 14:03:12 server sshd[9205]: Timeout before authentication for connection from 192.0.2.60 to 203.0.113.120, pid = 57844 Below are some common examples of log entries for failed certificate-based login attempts. There can sometimes be more than one problem with a certificate but in such cases only one error will get logged at a time. Expired or Not-yet-valid Certificate Certificates which have not yet become valid or which have already expired get a log entry as to the reason and include the offending certificate's id string and serial number as well as the fingerprint of the signing certificate authority. Below is one which is not yet valid being rejected by the server, perhaps at a future date it will be valid but that information is in the certificate, not the log. Aug 7 19:02:52 server sshd-session[96551]: error: Refusing certificate ID "fedcb" serial=9 signed by ED25519 CA SHA256:4ZyxpgCaw3Y8wz91ajLWARibUGfwyuOrftt2wermMJE: Certificate invalid: not yet valid Next is an expired certificate which has been rejected. Aug 7 19:02:53 server sshd-session[99417]: error: Refusing certificate ID "gfedc" serial=5 signed by ED25519 CA SHA256:4ZyxpgCaw3Y8wz91ajLWARibUGfwyuOrftt2wermMJE: Certificate invalid: expired If a certificate is not yet valid, then the log entry will say that explicitly and the trailing text on the line will be "Certificate invalid: not yet valid". The old style log entries for an expired certificate and for a certificate which has not yet become valid have much less information. Apr 5 14:33:31 server sshd-session[88953]: error: Certificate invalid: expired Apr 5 14:32:41 server sshd-session[51695]: error: Certificate invalid: not yet valid The improved logging arrived with OpenSSH 10.1 Valid Certificate but Invalid Principal Like with other reasons for rejecting certificates, when the account and allowed principals do not match the account logging in, information is logged about which certificate was tried and failed. Here is an entry showing an invalid principal: Aug 7 19:02:54 server sshd-session[12928]: error: Refusing certificate ID "edcba" serial=33 signed by ED25519 CA SHA256:4lngxTgobhprXfPY/p+2HNMBg6PxNDaOWrvei6kScrM: Certificate invalid: name is not a listed principal The old style log entry from OpenSSH 10.0 and earlier provides much less information about the certificate in question: Apr 5 14:34:56 server sshd-session[66692]: error: Certificate invalid: name is not a listed principal If the client closes the connection on purpose, there may be some information in the connection closed entry but that will be a different log entry. Valid Certificate but Invalid Source Address If the certificate is limited to connecting from specific addresses or host names, the log will complain if the connection comes from a different address or host and identify the incorrect source address. The information will be split across two log entries, which might or might not be adjacent depending on how busy the server is. Here is an entry showing an invalid source address: Aug 7 19:12:08 server sshd-session[29034]: cert: Authentication tried for fred with valid certificate but not from a permitted source address (198.51.100.135). Aug 7 19:12:08 server sshd-session[29034]: error: Refusing certificate ID "ihgfe" serial=8 signed by ED25519 CA SHA256:4ZyxpgCaw3Y8wz91ajLWARibUGfwyuOrftt2wermMJE: Refused by certificate options Prior to OpenSSH 10.1, the second entry had much less information about the offending certificate. Apr 5 14:57:47 server sshd-session[78381]: cert: Authentication tried for fred with valid certificate but not from a permitted source address (198.51.100.135). Apr 5 14:57:47 server sshd-session[78381]: error: Refused by certificate options However, it will not be possible to identify the specific certificate directly with that old style log entry. Logging SFTP File Transfers SFTP file transfers can be logged using LogLevel INFO or VERBOSE. The log level for the SFTP server can be set in sshd_config(5) separately from the general SSH server settings. Subsystem internal-sftp -l INFO By default the SFTP messages will also end up in auth.log but it is possible to filter these messages to their own file by reconfiguring the system logger, usually rsyslogd(8) or syslogd(8). Sometimes this is done by changing the log facility code from the default of AUTH. Available options are LOCAL0 through LOCAL7, plus, less usefully, DAEMON and USER. Subsystem internal-sftp -l INFO -f LOCAL6 If new system log files are assigned, it is important to remember them in the log rotation, too. Again, the Match directive can be use to change the log level just for certain connections. The following log excerpts are generated from using the log level INFO. A session starts with an open and end with a close. The number in the brackets is the process id for the SFTP session and is the only way to follow a session through the logs. Oct 22 11:59:45 server internal-sftp[4929]: session opened for local user fred from [192.0.2.33] ... Oct 22 12:09:10 server internal-sftp[4929]: session closed for local user fred from [192.0.2.33] Here is an SFTP upload of a small file of 928 bytes named foo to the home directory for user 'fred' when logging the SFTP at log level INFO. Oct 22 11:59:50 server internal-sftp[4929]: open "/home/fred/foo" flags WRITE,CREATE,TRUNCATE mode 0664 Oct 22 11:59:50 server internal-sftp[4929]: close "/home/fred/foo" bytes read 0 written 928 And a directory listing in the same session in the directory /var/www. Oct 8 07:28:25 server internal-sftp[34984]: opendir "/" Oct 8 07:28:25 server internal-sftp[34984]: closedir "/" Oct 8 07:28:26 server internal-sftp[34984]: opendir "/" Oct 8 07:28:26 server internal-sftp[34984]: closedir "/" Oct 8 07:28:27 server internal-sftp[34984]: opendir "/var/" Oct 8 07:28:27 server internal-sftp[34984]: closedir "/var/" Oct 8 07:28:27 server internal-sftp[34984]: opendir "/var/www/" Oct 8 07:28:27 server internal-sftp[34984]: closedir "/var/www/" And lastly here is a download of the same small 928-byte file called foo from the home directory for the user 'fred'. Oct 22 12:08:03 server internal-sftp[4929]: open "/home/fred/foo" flags READ mode 0666 Oct 22 12:08:03 server internal-sftp[4929]: close "/home/fred/foo" bytes read 928 written 0 Successful transfers will be noted by a close message. Attempts to download (open) files that do not exist will be followed by a sent status No such file message on a line of its own instead of a close. Files that exist but that the user is not allowed to read will create a sent status Permission denied message. Logging Chrooted SFTP Logging with the built-in sftp-subsystem inside a chroot jail, defined by ChrootDirectory, needs a ./dev/log node to exist inside the jail. This can be done by having the system logger such as syslogd(8) add additional log sockets inside the chrooted directory when starting up. On some systems that is as simple as adding more flags, like "-u -a /chroot/dev/log", in /etc/rc.conf.local or whatever the equivalent startup script may be. Here is an example of an SFTP login with password to a chroot jail using log level DEBUG3 for the SFTP-subsystem. The log shows a file upload: Apr 5 15:08:41 server sshd-session[31716]: Accepted password for fred from 198.51.100.135 port 39238 ssh2 Apr 5 15:08:41 server internal-sftp[55598]: session opened for local user fred from [198.51.100.135] Apr 5 15:08:41 server internal-sftp[55598]: received client version 3 Apr 5 15:08:41 server internal-sftp[55598]: debug1: request 1: limits Apr 5 15:08:41 server internal-sftp[55598]: debug3: request 2: realpath Apr 5 15:08:41 server internal-sftp[55598]: realpath "." Apr 5 15:08:41 server internal-sftp[55598]: debug1: request 2: sent names count 1 Apr 5 15:08:52 server internal-sftp[55598]: debug3: request 3: open flags 26 Apr 5 15:08:52 server internal-sftp[55598]: open "/home/fred/foo" flags WRITE,CREATE,TRUNCATE mode 0664 Apr 5 15:08:52 server internal-sftp[55598]: debug1: request 3: sent handle 0 Apr 5 15:08:52 server internal-sftp[55598]: debug1: request 5: write "/home/fred/foo" (handle 0) off 0 len 33 Apr 5 15:08:52 server internal-sftp[55598]: debug3: request 5: sent status 0 Apr 5 15:08:52 server internal-sftp[55598]: sent status Success Apr 5 15:08:52 server internal-sftp[55598]: debug3: request 4: close handle 0 Apr 5 15:08:52 server internal-sftp[55598]: close "/home/fred/foo" bytes read 0 written 33 Apr 5 15:08:52 server internal-sftp[55598]: debug3: request 4: sent status 0 Apr 5 15:08:52 server internal-sftp[55598]: sent status Success Remember that SFTP is a separate subsystem and that like the file creation mode, the log level and log facility are set separately from the SSH server in sshd_config(5), for example: Subsystem internal-sftp -l ERROR Logging Stability of Client Connectivity When ClientAliveInterval is set in the server's configuration, the server makes periodic probes of the clients which have established connections. At normal log levels, these are not noted in the log until something goes wrong. If the ClientAliveInterval is exceeded more times in a row than allowed by ClientAliveCountMax the client is officially declared disconnected and the connection dropped. At the default log level of INFO a brief message is logged, identifying the client which has been dropped. Sep 6 14:42:08 server sshd[83709]: packet_write_poll: Connection from 192.0.2.97 port 57608: Host is down At log level DEBUG, the client's responses to the polls will be logged by the server showing that the session is still connected. Sep 6 14:27:52 server sshd[9075]: debug1: Got 100/147 for keepalive Log level DEBUG2 and DEBUG3 will give even more information about the connection. However, even at log level DEBUG3, the specific client being polled will not be identified directly in the log messages and will have to be inferred from the process id of the daemon if such information is needed. Sep 6 14:30:59 server sshd[73960]: debug2: channel 0: request keepalive@openssh.com confirm 1 Sep 6 14:30:59 server sshd[73960]: debug3: send packet: type 98 Sep 6 14:30:59 server sshd[73960]: debug3: receive packet: type 100 Sep 6 14:30:59 server sshd[73960]: debug1: Got 100/22 for keepalive Again, when the ClientAliveCountMax is exceeded, the connection is broken after the final failure of the client to respond. Here is how that looks with the log level set to DEBUG2. Sep 6 14:17:55 server sshd[15780]: debug2: channel 0: request keepalive@openssh.com confirm 1 Sep 6 14:17:55 server sshd[15780]: debug1: Got 100/22 for keepalive Sep 6 14:18:37 server sshd[15780]: debug2: channel 0: request keepalive@openssh.com confirm 1 Sep 6 14:18:37 server sshd[15780]: packet_write_poll: Connection from 192.0.2.97 port 57552: Host is down Sep 6 14:18:37 server sshd[15780]: debug1: do_cleanup Sep 6 14:18:37 server sshd[48675]: debug1: do_cleanup Sep 6 14:18:37 server sshd[48675]: debug1: session_pty_cleanup: session 0 release /dev/ttyp0 The directives ClientAliveInterval and ClientAliveCountMax normally apply to all clients connecting to the server. However, they can be used inside a Match block and thus applied only to specific connections. Logging Revoked Keys If the RevokedKeys directive is used to point to a list of public keys that have been revoked, sshd(8) will make a log entry when access is attempted using a revoked key. The entry will be the same whether a plaintext list of public keys is used or if a binary Key Revocation List (KRL) has been generated. If password authentication is allowed, and the user tries it, then after the key authentication fails there will be a record of password authentication. Mar 14 20:36:40 server sshd-session[29235]: error: Authentication key RSA SHA256:jXEPmu4thnubqPUDcKDs31MOVLQJH6FfF1XSGT748jQ revoked by file /etc/ssh/ssh_revoked_keys ... Mar 14 20:36:45 server sshd-session[29235]: Accepted password for fred from 192.0.2.10 port 59967 ssh2 If password authentication is not allowed, sshd(8) will close the connection as soon as the key fails. Mar 14 20:38:27 server sshd-session[29163]: error: Authentication key RSA SHA256:jXEPmu4thnubqPUDcKDs31MOVLQJH6FfF1XSGT748jQ revoked by file /etc/ssh/ssh_revoked_keys ... The account trying the revoked key remains a mystery though. So it will be necessary to try to look up the key by its fingerprint from your archive of old keys using ssh-keygen -lf and read the key's comments. Although if a valid account cancels the connection without trying a password after the key attempt fails, the usual message will still be posted to the log and that will show the login name. Mar 14 20:44:04 server sshd-session[14352]: Connection closed by authenticating user fred 192.0.2.237 port 55051 [preauth] ... That may be enough to provide some clue and allow a bit of log filtering with a short AWK script, if the messages are all in the same log file. $ awk '/revoked by file/ { pid[$5]++; key[$5]=$9; hash[$5]=$10; next; } pid[$5] && /closed by authenticating user/ { print key[$5], hash[$5], $10, $11; delete key[$5]; delete hash[$5]; delete pid[$5]; }' /var/log/authlog Similarly, if the client makes no attempt at logging in and just times out, the message will say just that. Mar 18 21:40:25 server sshd[9942]: fatal: Timeout before authentication for connection from 198.51.100.236 to 203.0.113.75, pid = 67611 ... On the client side, no warning or error will be given if a revoked key is tried. It will just fail and the next key or method will be tried. Brute force and Hail Mary attacks It’s fairly common to see failed login attempts almost as soon as the server is connected to the net. Brute force attacks, where one machine hammers on a few accounts trying to find a valid password, are becoming rare. In part this is because packet filters, like NFables for Linux and PF for the BSDs, can limit the number and rate of connection attempts from a single host. The server configuration directive MaxStartups can limit the number of simultaneous, unauthenticated connections. ... Apr 4 16:45:34 server sshd-session[96275]: Failed password for root from 103.42.56.121 port 48294 ssh2 ... Apr 4 16:46:58 server sshd-session[96275]: Failed password for root from 103.42.56.121 port 48294 ssh2 ... Apr 4 16:47:04 server sshd-session[96275]: Failed password for root from 103.42.56.121 port 48294 ssh2 Apr 4 16:47:05 server sshd-session[96275]: Connection closed by authenticating user root 103.42.56.121 port 48294 [preauth] ... Prior to 9.8, it was slightly different: ... Mar 18 18:54:44 server sshd[54939]: Failed password for root from 201.179.249.231 port 52404 ssh2 Mar 18 18:54:48 server sshd[54939]: Failed password for root from 201.179.249.231 port 52404 ssh2 Mar 18 18:54:49 server sshd[54939]: Failed password for root from 201.179.249.231 port 52404 ssh2 Mar 18 18:54:49 server sshd[54939]: error: maximum authentication attempts exceeded for root from 201.179.249.231 port 52404 ssh2 [preauth] Mar 18 18:54:49 server sshd[54939]: Disconnecting authenticating user root 201.179.249.231 port 52404: Too many authentication failures [preauth] ... Note the "authenticating user" is present in the logs from OpenSSH 7.5 and onward when a valid user name is attempted. When an invalid user name is attempted, that is written too. ... Apr 3 10:25:59 server sshd-session[78513]: Invalid user ad1tz from 103.42.56.121 port 55778 Apr 3 10:26:00 server sshd-session[78513]: Received disconnect from 103.42.56.121 port 55778:11: Bye Bye [preauth] Apr 3 10:26:00 server sshd-session[78513]: Disconnected from invalid user ad1tz 103.42.56.121 port 55778 [preauth] ... Again, in versions prior to 9.8, the process name is different as is the closing message. ... Mar 18 18:55:05 server sshd[38594]: Invalid user ubnt from 201.179.249.231 port 52471 Mar 18 18:55:05 server sshd[38594]: Failed password for invalid user ubnt from 201.179.249.231 port 52471 ssh2 Mar 18 18:55:09 server sshd[38594]: error: maximum authentication attempts exceeded for invalid user ubnt from 201.179.249.231 port 52471 ssh2 [preauth] Mar 18 18:55:09 server sshd[38594]: Disconnecting invalid user ubnt 201.179.249.231 port 52471: Too many authentication failures [preauth] ... The way to deal with brute force attacks coming from a single machine or network is to customize the server host’s packet filter to limit the attacks or even temporarily block machines that overload the maximum number or rate of connections. Optionally, one should also contact the attacker’s net block owner with the IP address and exact date and time of the attacks. A kind of attack common at the time of this writing is one which is distributed over a large number of compromised machines, each playing only a small role in attacking the server. To deal with Hail Mary attacks, contact the attacker’s net block owner. A form letter with a cut-and-paste excerpt from the log is enough if it gives the exact times and addresses. Alternately, teams of network or system administrators can work to pool data to identify and blacklist the compromised hosts participating in the attack. Failed None For Invalid User The SSH protocol specifies a number of possible authentication methodshttps://datatracker.ietf.org/doc/html/rfc4252#section-5.2 RFC 4252: The Secure Shell (SSH) Authentication Protocol 2006 IETF 2021-08-10 . The methods password, keyboard-interactive, and publickey are fairly common. A lesser known authentication method is none, which will only succeed if the server requires no further authentication such as if PermitEmptyPassword is set and the account does not actually have a passwordhttps://marc.info/?l=openbsd-misc&m=162858437916966&w=2 Tucker Darren OpenBSD-Misc 2021-08-10 2021-08-10 Re: ssh authlog: Failed none for invalid user OpenBSD. Some SSH clients including OpenSSH's start by asking for none authentication and then use the list of remaining possible authentication methods to decide what to do next if that doesn't work. ... Aug 10 19:09:05 server sshd[93126]: Failed none for invalid user admin from 125.64.94.136 port 27586 ssh2 ... So in other words, that is a brute force attack trying the none authentication method. It is an attack which will only get into accounts which have been explicitly set with an empty password and furthermore have also been set up specifically to allow access by having both the authentication method of none and the PermitEmptyPasswords configuration directive enabled on the server. Most brute force attacks try only password authentication, and some of those even check for the password method and then give up if it is not available. Other attackers may just hammer away pointlessly even if the method is not available. Connections Seemingly From 127.0.0.1, ::1, or Other localhost Addresses When the SSH server is accessed via a reverse tunnel to another machine, the incoming connections will appear to be from the localhost address, which is usually 127.0.0.1 or ::1. Mar 23 14:16:16 server sshd[9265]: Accepted password for fred from 127.0.0.1 port 40426 ssh2 If the port at the other end of the reverse tunnel is publicly accessible, it will be probed and possibly attacked. Because of the reverse tunnel, the attacks will then also appear to be coming from the server's own loopback address: Mar 23 14:20:17 server sshd-session[5613]: Invalid user cloud from ::1 port 57404 Mar 23 14:20:21 server sshd-session[5613]: Failed password for invalid user cloud from ::1 port 57404 ssh2 Mar 23 14:20:26 server sshd-session[5613]: Failed password for invalid user cloud from ::1 port 57404 ssh2 Mar 23 14:20:35 server sshd-session[5613]: Connection closed by invalid user cloud ::1 port 57404 [preauth] Like in the other examples, the disconnection message was different prior to 9.8: ... Mar 23 14:20:32 server sshd[5613]: Failed password for invalid user cloud from ::1 port 57404 ssh2 Mar 23 14:20:35 server sshd[5613]: Connection closed by invalid user cloud ::1 port 57404 [preauth] ... Therefore the usual countermeasures like SSHGuard or Fail2Ban or other similar intrusion detection systems cannot be used because the localhost address is used by the tunnel for all login attempts, regardless of their real origins. A partial solution would be to bind the incoming connections to a different IP address. The loopback interface would need an additional permanent address, an alias, for that. That alias could then be assigned when establishing the reverse tunnel: $ ssh -R 2022:127.2.2.1:22 fred@vps.example.org Thus it would designate the source address for all logins coming in over that tunnel. So, in that way, the alias would then show up in the logs instead of the default loopback address when that reverse tunnel is used: Mar 23 18:00:13 server sshd-session[8525]: Invalid user cloud from 127.2.2.1 port 17271 Mar 23 18:00:15 server sshd-session[8525]: Failed password for invalid user cloud from 127.2.2.1 port 17271 ssh2 Mar 23 18:00:19 server sshd-sesssion[8525]: Failed password for invalid user cloud from 127.2.2.1 port 17271 ssh2 Mar 23 18:01:23 server sshd-session[8525]: Failed password for invalid user cloud from 127.2.2.1 port 17271 ssh2 Mar 23 18:01:26 server sshd-session[8525]: Connection closed by invalid user cloud 127.2.2.1 port 17271 [preauth] If the ports do not need to be available to the open Internet, a full solution would be just to ensure that they are not accessible from the outside. This would be done by not using the -g option on the client when making the reverse tunnel or else by setting the GatewayPorts directive in sshd_config(5) back to the default of no, or both. The system's built-in packet filter can also be used. Then, even with the forwarded ports closed off from the outside, the ProxyJump option can still be used to skip through the jump host and use the setup for SSH access. However, since it is sometimes necessary that these ports be accessible to the outside world, this approach is not always an option. Client Logging The OpenSSH client normally sends log information to stderr. The -y option can be used to send output to the system logs, managed by syslogd(8) or something similar. Client log verbosity can be increased or decreased by changing the LogLevel directive and the log facility changed with the SyslogFacility directive in ssh_config(5). Both require use of the -y run-time option and do nothing without it. Alternatively, instead of using the -y option, using the -E option sends log output to a designated file instead of stderr. Working with the system logs or separate log files are something which can be useful when running ssh(1) in automated scripts. Below is an example of a connection to an interactive shell with the normal level of client logging: $ ssh -l fred server.example.org fred@server.example.org‘s password: Last login: Thu Jan 27 13:21:57 2011 from 192.168.11.1 The same connection at the first level of verbosity gives lots of debugging information, 90 lines in all. $ ssh -v -l fred server.example.org debug1: OpenSSH_10.6, LibreSSL 4.4.0 debug1: Reading configuration data /etc/ssh/ssh_config debug1: no pubkey loaded from /home/fred/.ssh/id_rsa debug1: identity file /home/fred/.ssh/id_rsa type -1 debug1: no identity pubkey loaded from /home/fred/.ssh/id_rsa debug1: no pubkey loaded from /home/fred/.ssh/id_ecdsa debug1: identity file /home/fred/.ssh/id_ecdsa type -1 debug1: no identity pubkey loaded from /home/fred/.ssh/id_ecdsa debug1: no pubkey loaded from /home/fred/.ssh/id_ecdsa_sk debug1: identity file /home/fred/.ssh/id_ecdsa_sk type -1 debug1: no identity pubkey loaded from /home/fred/.ssh/id_ecdsa_sk debug1: no pubkey loaded from /home/fred/.ssh/id_ed25519 debug1: identity file /home/fred/.ssh/id_ed25519 type -1 debug1: no identity pubkey loaded from /home/fred/.ssh/id_ed25519 debug1: no pubkey loaded from /home/fred/.ssh/id_ed25519_sk debug1: identity file /home/fred/.ssh/id_ed25519_sk type -1 debug1: no identity pubkey loaded from /home/fred/.ssh/id_ed25519_sk debug1: no pubkey loaded from /home/fred/.ssh/id_mldsa44_ed25519 debug1: identity file /home/fred/.ssh/id_mldsa44_ed25519 type -1 debug1: no identity pubkey loaded from /home/fred/.ssh/id_mldsa44_ed25519 debug1: Connecting to server.example.org [198.51.100.20] port 22. debug1: Connection established. debug1: Local version string SSH-2.0-OpenSSH_10.6 debug1: Remote protocol version 2.0, remote software version OpenSSH_10.6 debug1: compat_banner: match: OpenSSH_10.6 pat OpenSSH* compat 0x04000000 debug1: Authenticating to server.example.org:22 as 'fred' debug1: load_hostkeys: fopen /home/fred/.ssh/known_hosts2: No such file or directory debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts: No such file or directory debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts2: No such file or directory debug1: SSH2_MSG_KEXINIT sent debug1: SSH2_MSG_KEXINIT received debug1: kex: algorithm: mlkem768x25519-sha256 debug1: kex: host key algorithm: ssh-ed25519 debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none debug1: expecting SSH2_MSG_KEX_ECDH_REPLY debug1: SSH2_MSG_KEX_ECDH_REPLY received debug1: Server host key: ssh-ed25519 SHA256:agRZDZn8awhQ+tc19Ip/whkHoV33Ln2REO7v1kcYbdU debug1: load_hostkeys: fopen /home/fred/.ssh/known_hosts2: No such file or directory debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts: No such file or directory debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts2: No such file or directory debug1: Host 'server.example.org' is known and matches the ED25519 host key. debug1: Found key in /home/fred/.ssh/known_hosts:1 debug1: ssh_packet_send2_wrapped: resetting send seqnr 3 debug1: rekey out after 134217728 blocks debug1: SSH2_MSG_NEWKEYS sent debug1: Sending SSH2_MSG_EXT_INFO debug1: expecting SSH2_MSG_NEWKEYS debug1: ssh_packet_read_poll2: resetting read seqnr 3 debug1: SSH2_MSG_NEWKEYS received debug1: rekey in after 134217728 blocks debug1: SSH2_MSG_EXT_INFO received debug1: kex_ext_info_client_parse: server-sig-algs=<ssh-ed25519,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,sk-ssh-ed25519@openssh.com,sk-ecdsa-sha2-nistp256@openssh.com,rsa-sha2-512,rsa-sha2-256,ssh-mldsa44-ed25519,webauthn-sk-ecdsa-sha2-nistp256@openssh.com> debug1: kex_ext_info_check_ver: publickey-hostbound@openssh.com=<0> debug1: kex_ext_info_check_ver: ping@openssh.com=<0> debug1: kex_ext_info_check_ver: agent-forward=<0> debug1: SSH2_MSG_SERVICE_ACCEPT received debug1: SSH2_MSG_EXT_INFO received debug1: kex_ext_info_client_parse: server-sig-algs=<ssh-ed25519,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,sk-ssh-ed25519@openssh.com,sk-ecdsa-sha2-nistp256@openssh.com,rsa-sha2-512,rsa-sha2-256,ssh-mldsa44-ed25519,webauthn-sk-ecdsa-sha2-nistp256@openssh.com> debug1: Authentications that can continue: publickey,password,keyboard-interactive debug1: Next authentication method: publickey debug1: Will attempt key: /home/fred/.ssh/id_rsa debug1: Will attempt key: /home/fred/.ssh/id_ecdsa debug1: Will attempt key: /home/fred/.ssh/id_ecdsa_sk debug1: Will attempt key: /home/fred/.ssh/id_ed25519 debug1: Will attempt key: /home/fred/.ssh/id_ed25519_sk debug1: Will attempt key: /home/fred/.ssh/id_mldsa44_ed25519 debug1: Trying private key: /home/fred/.ssh/id_rsa debug1: Trying private key: /home/fred/.ssh/id_ecdsa debug1: Trying private key: /home/fred/.ssh/id_ecdsa_sk debug1: Trying private key: /home/fred/.ssh/id_ed25519 debug1: Trying private key: /home/fred/.ssh/id_ed25519_sk debug1: Trying private key: /home/fred/.ssh/id_mldsa44_ed25519 debug1: Next authentication method: keyboard-interactive debug1: Authentications that can continue: publickey,password,keyboard-interactive debug1: Next authentication method: password Authenticated to server.example.org ([198.51.100.20]:22) using "password". debug1: channel 0: new session [client-session] (inactive timeout: 0.000000) debug1: Requesting no-more-sessions@openssh.com debug1: Entering interactive session. debug1: pledge: filesystem debug1: client_input_global_request: rtype hostkeys-00@openssh.com want_reply 0 debug1: client_input_hostkeys: searching /home/fred/.ssh/known_hosts for server.example.org / (none) debug1: client_input_hostkeys: searching /home/fred/.ssh/known_hosts2 for server.example.org / (none) debug1: client_input_hostkeys: hostkeys file /home/fred/.ssh/known_hosts2 does not exist debug1: client_input_hostkeys: no new or deprecated keys from server debug1: pledge: fork debug1: client_input_channel_req: channel 0 rtype keepalive@openssh.com reply 1 debug1: client_input_channel_req: channel 0 rtype keepalive@openssh.com reply 1 debug1: client_input_channel_req: channel 0 rtype keepalive@openssh.com reply 1 Last login: Sat Mar 14 21:31:33 2015 from 192.0.2.111 ... The same login with the maximum of verbosity, -vvv, gives around 239 lines of debugging information. The following will save the debugging information separately in a file, ssh-client.log, in the home directory. $ ssh -vvv -E ~/ssh-client.log -l fred somehost.example.org Capturing Client Debugging Information Together with the Session Contents Regular pipes and redirects work only with stdout so that if -E is not used to capture debugging output the output on stderr must instead be sent to stdout if one is going to capture it at the same time as the actual session. That is done with an extra redirect, 2>&1 to capture stderr. Mind the spaces, or lack of them. The following will capture both the session text and the debugging information to a file, ssh-client.log, in the home directory: $ ssh -vvv -l fred somehost.example.org 2>&1 | tee ~/ssh-output.log The tool tee(1) is like a T-pipe and sends output two directions, one to stdout and one to a file. Changing Client Debugging Levels At Runtime or On The Fly At run-time, when establishing a new connection, just use the -v option. $ sftp -v -o "IdentityFile=~/.ssh/weblog.key_ed25519" fred@server.example.org The debugging verbosity on the client can be increased just like on the server. $ sftp -vvv -o "IdentityFile=~/.ssh/weblog.key_ed25519" fred@server.example.org The extra information can be useful to see exactly what is being sent to or requested of the server. After the fact, once a connection is established, the escape sequences ~v and ~V can be used to increase or decrease, respectively, the verbosity on the fly. Through them it is possible to change the verbosity of the client in an established connection. When increasing the log level with ~v, the client raises its log level to VERBOSE, DEBUG, DEBUG2, and DEBUG3, in that order, if starting from the default of INFO. Conversely, when lowering the log level with ~V, the client will descend through ERROR, FATAL, to QUIET if starting from the default of INFO. Debugging and Troubleshooting The server logs are your best friend when troubleshooting. It may be necessary to turn up the log level there temporarily to get more information. It is then also necessary to turn them back to normal after things are fixed to avoid privacy problems or excessively use of disk space. For example, the SFTP-subsystem logging defaults to ERROR, reporting only errors. To track transactions made by the client, change the log level to INFO or VERBOSE: Subsystem internal-sftp -l INFO Caution. Again, operating with elevated logging levels would violate the privacy of users, in addition to filling a lot of disk space. Elevated logging levels should generally not be used in production once the changes are figured out. Even then, the elevated log messages should really be sent to a separate log file during the time they are being collected. By default, some systems send only the normal messages to the regular system log files and ignore the elevated messages. Some save all the messages by default. If the elevated system log messages are not showing up in any of the system logs, that may be the reason. Either way, check the system log configurations and make sure that the extra messages are only sent to a separate log file and not mixed in with the regular system logs. Change the configuration to do that, if necessary. Separating the extra messaging helps keep the logs tidy as well as protecting privacy. The system log settings are found in the system log daemon's configuration file, the exact name of which will vary depending on what is installed, but common ones are syslog.conf(5) and rsyslog.conf(5). Notice that the configuration below has the all messages including the more detailed VERBOSE, and on up through DEBUG3, messages for the AUTH facility going to a separate log file from the regular AUTH messages: $ grep '^auth\.' /etc/syslog.conf auth.info /var/log/authlog auth.* /var/log/authdebug See syslog(3) for the log facilities and log levels. It is best to limit the time the debugging information is collected to be as short as possible and to actively watch the logs while the extra information is being collected. However, if the more verbose logging is running for any length of time, and especially if it is left unattended even for a short while, be sure to remember to add the new log file to the log rotation schedule so that it cannot fill up the partition. Note, logs might be handled by other services on some systems. Check with the official documentation for your system to be sure. On the client side, Match blocks can help further by setting log levels for specific situations and avoid a situation where everything is logged intensely. The manual pages for OpenSSH are very well written and many times problems can be solved by finding the right section within the right manual page. At the very minimum, it is important to skim through the four main manual pages for both the programs and their configuration to become familiar with at least the section headings. ssh(1) ssh_config(5) sshd(8) sshd_config(5) Once the right section is found in the manual page, read it in detail and become familiar with its contents. The same goes for the other OpenSSH manual pages, depending on the activity. Be sure to use the version(s) of OpenSSH actually in use on your system(s) with their corresponding manual pages. When possible, use those that are installed on your system so as to avoid a mismatch. In some cases, the client and the server will be of different versions, so the manual pages for each must be looked up separately. It is also a good idea to review OpenSSH's release notes when a new version is published. With a few exceptions below, specific examples of troubleshooting are usually given in the cookbook section relevant to a particular activity. So, for example, sorting problems with authentication keys is done in the section on Public Key Authentication itself. Debugging a script, configuration or key that uses sudo(8) Usually log levels only need to be changed when writing and testing a script, a new configuration, some new keys, or all three at once. When working with sudo(8), it is especially important to see exactly what the client is sending so as to enter the right pattern into /etc/sudoers for safety. Using the lowest level of verbosity, the exact string being sent by the client to the remote server is shown in the debugging output: $ rsync -e "ssh -v -i /home/webmaint/.ssh/bkup_key -l webmaint" \ -a server.example.org:/var/www/ var/backup/www/ ... debug1: Sending command: rsync --server --sender -logDtpre.iLsfxCIvu . /var/www/ ... What sudoers then needs is something like the following, assuming account 'webmaint' is in the group 'webmasters': %webmasters ALL=(ALL) NOPASSWD: /usr/local/bin/rsync --server \ --sender -logDtpre.iLsfxCIvu . /var/www/ The same method can be used to debug new server configurations or key logins. Once things are set to run as needed, the log level settings can be lowered back to INFO for sshd(8) and to ERROR for internal-sftp. Additionally, once the script is left to run in fully automated mode, the client logging information can be set use the syslog(3) system module instead of stderr by using the -y option when ssh(1) is launched. Debugging a server configuration Running the server in debug mode provides a lot of information about the connection and a smaller amount about the server configuration. The server's debugging level (-d) can be raised once, twice (-dd) or thrice (-ddd). $ /usr/sbin/sshd -d Note that in this case the server does not detach and become a daemon, so it will terminate when the SSH connection terminates. The advantage is this collects the logs for one single session and only that session. The server must be started again in order to make a subsequent connection from the client. Though in some ways this is a hassle, it does make sure that session data is a unique set and not mixes of multiple sessions and thus possibly different configurations. Alternately, another option (-e) when debugging sends the debugging data to stderr instead in order to keep the system logs clean. It is also possible to log the debugging data from the system logs directly to a separate file and keep noise out of the system logs. Since OpenSSH 6.3, the option -E will append the debug data to a particular log file instead of sending it to the system log. This facilitates debugging live systems without cluttering the system logs. $ /usr/sbin/sshd -E /home/fred/sshd.debug.log -d In really old versions of OpenSSH, which are hopefully no longer in production, if you need to save output to a file while still viewing it live on the screen, you can use tee(1). $ /usr/sbin/sshd -ddd 2>&1 | tee /tmp/foo That will save output to the file foo by redirecting to stdout what sshd(8) sent to stderr and saving it to a file. This works with really old versions of OpenSSH, but the -E option above is preferable. If the server is remote and it is important to reduce the risk of getting locked out, the experiments on the configuration file can be done with a second instance of sshd(8) using a separate configuration file and listening to a high port until the settings have been tested. $ /usr/sbin/sshd -dd -p 22222 -f /home/fred/sshd_config.test If the OpenSSH server's configuration file is set to listen on odd ports or more than one port using the ListenAddress directive, then that can also be overriden so that the server listens on the extra port. $ /usr/sbin/sshd -o 'ListenAddress 0.0.0.0:22222' -dd \ -f /home/fred/sshd_config.test It is also possible to make an extended test (-T) of the configuration file first. If there is a syntax error, it will be reported, but remember that even sound configurations can still lock you out. $ /usr/sbin/sshd -T | sort | less -X The extended test mode can be used by itself, but it is also possible to specify custom parameters to use with -C to see what happens with specific kinds of connections. sshd(8) will the connection. It will then process the configuration file in light of the parameters passed to it and return output with the modified results. This is good for showing the results of any Match directives which might apply to the session. The variables user, host, and addr are the minimum required when passing specific connection parameters to sshd(8) for evaluation with the -C option . The following options will print out the configurations that will be applied when the user fred tries to log in to the host server.example.org from the address 192.0.2.15: $ /usr/sbin/sshd -T -C user=fred,host=server.example.org,addr=192.0.2.15 Two more parameters, laddr and lport, may also be passed. They refer to the server's IP number and port connected to. $ /usr/sbin/sshd -T -C user=fred,host=server.example.org,addr=192.0.2.15,laddr=192.0.2.2,lport=2222 And lastly, there is a parameter rdomain to specify the routing domain or routing table to try. Between those six variables it should be possible to describe any possible incoming connection. Adding Timestamps to Server Debugging Output The -d option increases the value for LogLevel but does not prepend any timestamps to the server's output. Timestamps can nevertheless be added on the fly by writing the debug logs to stderr using the -e option, redirecting stderr to stdout, and then piping that into ts(1), a timestamping utility. $ /usr/sbin/sshd -p 2222 -e -d 2>&1 | ts "%.T" | tee /tmp/sshd.log That will show the time, including microseconds. Mind the dot there before the letter 'T'. The options -i and -s for ts(1) can be relevant, too. The -i option will measure the relative time between the timestamps themselves. The -s option will show the time elapsed since the start of the program. Other custom date and time formats can be specified as per strftime(3), if needed. When ts(1) is not already present it can usually be added via the moreutils package. Although some systems, such as OpenBSD, include it in the base system. If it is not already installed and not available for installation, then Perl can be used on POSIX systems to prepend a timestamp to stdin: $ /usr/sbin/sshd -p 2222 -e -d 2>&1 \ | perl -p -e 'use POSIX strftime; print strftime("%F %T ", localtime)' \ | tee /tmp/sshd.log However, that only provides an accuracy of whole seconds. If more precision is needed, then Python3 can provide timestamps with precision down to the microsecond. $ /usr/sbin/sshd -p 2222 -e -d 2>&1 \ | python -c 'import datetime; import sys; [print (datetime.datetime.now(), flush=True) for line in sys.stdin]' \ | tee /tmp/sshd.log Note that the Python approach requires explicitly setting the buffer to flush automatically. Some non-POSIX shells can make use of process substitution to pipe stderr output into another program. $ /usr/sbin/sshd -p 2222 -e -d 2> >(ts) | tee /tmp/sshd.log | less -X There are even fancier approaches using IO redirection on extra file descriptors, but those are far outside the scope of this book. Debugging a client configuration Sometimes when debugging a server configuration it is necessary to track the client configurations settings, as well. Since OpenSSH 6.8, the -G option makes ssh(1) print its configuration after evaluating any Host and Match blocks and then exit without connecting. That allows viewing of the exact configuration options which will actually be used by the client for a particular session. $ ssh -G -l fred server.example.org | sort Client configuration is determined in three ways. The first is by run-time options, then by the account's own configuration file, or lastly the system-wide client configuration file. The priority is in that order. Whichever value is found first gets used. With sftp(1) the options are also passed to ssh(1). Invalid or Outdated Ciphers or MACs A proper client will show the details of the failure. For a bad Message Authentication Code (MAC), a proper client might show something like the following when trying to foist a bad MAC like hmac-md5-96 onto the server: no matching mac found: client hmac-md5-96 server umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1 And for a bad cipher, a proper client might show something like this when trying to foist an arcfour cipher on the server: no matching cipher found: client arcfour server chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com Sometimes when troubleshooting a problem with the client it is necessary to turn to the server logs. In OpenSSH 6.7 unsafe MACs were removed and in OpenSSH 7.2 unsafe ciphers were removed, but some third-party clients may still try to use them to establish a connection. In that case, the client might not provide much information beyond a vague message that the server unexpectedly closed the network connection. The server logs will, however, show what happened: fatal: no matching mac found: client hmac-sha1,hmac-sha1-96,hmac-md5 server hmac-sha2-512-etm@openssh.com,hmac-sha2-256-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-512,hmac-sha2-256,hmac-ripemd160 [preauth] More recent versions would show a simpler error for a bad MAC. fatal: Unable to negotiate with 192.0.2.37 port 55044: no matching MAC found. Their offer: hmac-md5-96 [preauth] A bad cipher would be reported like this: fatal: Unable to negotiate with 192.0.2.37 port 55046: no matching cipher found. Their offer: arcfour [preauth] The error message in the server log might not say which MACs or ciphers are actually available. For those, the extended test mode can be used to show the server settings and, in particular, the MACs or ciphers allowed. In its most basic usage the extended test mode would just be -T, as in /usr/sbin/sshd -T | grep -E 'cipher|macs' with no other options. For more details and options, see the previous section on "Debugging a server configuration" above. One solution there is to upgrade the client to one that can handle the right ciphers and MACs. Another option is to switch to a different client, one that can handle the modern ciphers or MACs. See also the configuration directive WarnWeakCrypto in the manual page. It has been around since 10.1. The default for WarnWeakCrypto is 'yes'. Debugging Key-Based Authentication The most common causes of failure for public key authentication seem to be for either of two reasons: Mangling the public key on the way to getting it into authorized_keys on the server Incorrect permissions for the files and directories involved, ether on the client or the server. These are the directory for the keys, usually ~/.ssh/, or its parent directories, or the authorized_keys file, or the private key itself. As of the time of this writing, it looks like pretty much every failure of key-based authentication described on mailing lists and forums is solved by addressing either or both of those two situations. So, when encountering the error message Permission denied (publickey,keyboard-interactive), or similar, see the section on Public Key Authentication. Also be sure to read the manual page for sshd(8) and its section on authorized keys. Usually, though not always, it will be obvious when the private key's permissions are incorrect: $ ssh -i ~/.ssh/fred-193.ed25519 fred@192.0.2.193 @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ @ WARNING: UNPROTECTED PRIVATE KEY FILE! @ @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ Permissions 0664 for '/home/fred/.ssh/fred-193.ed25519' are too open. It is required that your private key files are NOT accessible by others. This private key will be ignored. Load key "/home/fred/.ssh/fred-193.ed25519": bad permissions In addition mangled public keys in the authorized keys file and to incorrect permissions, a very rare third case is if the public and private key files don't match and are not from the same key pair. As mentioned in the section on Public Key Authentication, the public and private keys need to be parts of the same key pair so that they match. That is because even before the SSH client uses private key cryptographically, it looks at the file name of the proposed private key and then sends the public key matching that same name, if it exists, to the server. If that public key does not match the public key on the server in authorized_keys then the connection will be denied with the error Permission denied (publickey,keyboard-interactive) or similar. That alone is a very good reason to give unique descriptive file names when creating keys. Note that as mentioned above there are usually other causes to that same error message besides having mismanaged the public key file on the client machine. The file names for both parts of each key pair have to be kept synchronized so that the names match. As for a solution, the way out in the long term is to stay organized and carefully manage the keys and their file names. It can be solved on the short term by deleting the offending public key file and then using the private key to regenerate a new one, replacing the incorrect public key under that name. Again, this is an unusual edge case and not a common cause of that error. SSH Too Many Authentication Failures When there are multiple keys in the authentication agent, the client will try them against the server in the same order in which they were added to the agent. However, on odd or older agents they are returned in an unpredictable order. Newer versions allow the client to query the server to see if a particular key is acceptable. This is set by the PubkeyAuthOptions and is counted separately from the MaxAuthTries directive. When the former are used up, then the additional tries count against the latter. Either way, if the client happens to cycle through enough of the wrong keys first and hits the server's MaxAuthTries limit before finding the right key, the server will naturally break off the connection with an error message on the server about too many authentication failures: Disconnecting authenticating user fred 203.0.113.110 port 35893: Too many authentication failures [preauth] Or in earlier versions: "Received disconnect from 203.0.113.110 port 22:2: Too many authentication failures Authentication failed." When that happens, the client side will show this error: Received disconnect from server.example.org port 22:2: Too many authentication failures Disconnected from server.example.org port 22 With increased verbosity, the specific keys tested and rejected will also be shown by the client. Newer clients will include the key's comment, if available, and not just the fingerprint. The keys get tried first from the agent in the order in which they were loaded into the agent, then from : $ ssh -v 203.0.113.110 . . . hdebug1: Offering public key: a key for fred ED25519 SHA256:7e5qkSHxSp8xitlM4JP4dOeCdnQn Vt9q6GPaB0+IrSM agent debug1: Authentications that can continue: publickey,keyboard-interactive debug1: Offering public key: fred's key ED25519 SHA256:8U7Fy6nJxbxfAu0K8WXZV0WN80t/ 4lb/f9bqObw0isE agent debug1: Authentications that can continue: publickey,keyboard-interactive debug1: Offering public key: one more comment ED25519 SHA256:F61Sx2t4BmoHASuMCVrBd8C30A7S QdGPwgfb7ZGWZf0 agent debug1: Authentications that a publickey,keyboard-interactive debug1: Offering public key: fred@desk.example.org ED25519 SHA256:TMfO5S6oBQ0Cu273yPp9+7carGP926kRfg6iS435cmU agent debug1: Authentications that can continue: publickey,keyboard-interactive debug1: Offering public key: yet one more key ED25519 SHA256:36tMBWe4kX9hyVLZaxyjm2pp1TMFbRw0+O1xWOU5Fgk agent debug1: Authentications that can continue: publickey,keyboard-interactive debug1: Offering public key: the last straw ED25519 SHA256:gNZd1TF48UgEUgoYFNXop9tEhO53If1qiDqZyKbRT4U agent Received disconnect from 203.0.113.110 port 2222:2: Too many authentication failures Disconnected from 203.0.113.110 port 2222 Each key in the agent also gets an annotation which says whether or not the key file was supplied by the user, either in the configuration file or as a run-time argument. For the former, it will marked with 'agent' and for the latter two as 'explicit'. The client prefers keys that are currently in the agent in the order in which they were added to the agent. Then it will try the one(s) added via run-time arguments, in the order in which they are supplied. Then, lastly, it will try the one(s) specified in the client configuration file. Or in earlier versions the file name used to load the key will be listed by the client in verbose mode, but the order will be less predictable: $ ssh -v 203.0.113.110 ... debug1: SSH2_MSG_SERVICE_ACCEPT received debug1: Authentications that can continue: publickey,keyboard-interactive debug1: Next authentication method: publickey debug1: Offering RSA public key: /home/fred/.ssh/key.06.rsa debug1: Authentications that can continue: publickey,keyboard-interactive debug1: Offering RSA public key: /home/fred/.ssh/key.02.rsa debug1: Authentications that can continue: publickey,keyboard-interactive debug1: Offering RSA public key: /home/fred/.ssh/key.03.rsa debug1: Authentications that can continue: publickey,keyboard-interactive debug1: Offering RSA public key: /home/fred/.ssh/key.04.rsa debug1: Authentications that can continue: publickey,keyboard-interactive debug1: Offering RSA public key: /home/fred/.ssh/key.01.rsa debug1: Authentications that can continue: publickey,keyboard-interactive debug1: Offering RSA public key: /home/fred/.ssh/key.05.rsa Received disconnect from port 22:2: Too many authentication failures Authentication failed. In the older clients, the client prefers keys that were specified in the configuration and are also currently in the agent. Then it will try try them in the order in which they were supplied. [3] https://lists.mindrot.org/pipermail/openssh-unix-dev/2019-April/037700.html IdentityFile vs IdentitiesOnly 2019-04-01 2019-04-04 openssh-unix-dev Tucker Darren There are two solutions if you see the Too many authentication failures error: One way around this error is to remove keys from the agent one at a time using ssh-add(1) with the -d option until there is just the right key left. Refer to each key by its file system path, for example: ssh-add -d ~/.ssh/some.key.rsa Because the private key to be removed is looked up in the agent based on the corresponding public key both files must exist. Without matching a public key file, the private key cannot be removed individually from the authentication agent. Instead the whole lot may be removed all at once using the -D option. However, is not always practical to do either when many remote systems are used frequently and the agent needs to be kept well-stocked. This is probably not the most practical way. Another way around this error, and probably the most practical method, is to limit the client to trying only a specific key using the IdentitiesOnly configuration directive in conjunction with the IdentityFile configuration directive. The latter points explicitly to the right key. Both can be added either as run-time options or in the client's configuration file. As a runtime option, they can be used like this: $ ssh -o IdentitiesOnly=yes -i ~/.ssh/server14.example.org.rsa -l fred server14.example.org Those settings can be stored permanently by adding something like the following IdentitiesFile and IdentitiesOnly directives to the client configuration file instead. Host server14 server14.example.org HostName server14.example.org IdentitiesOnly yes IdentityFile /home/fred/.ssh/server14.example.org.rsa User fred In that way, the server could be reached with either the short name or the fully qualified domain name, whatever names are listed under the Host stanza. $ ssh server14 Remember that options are selected from the client configuration file on a first-match basis. Because the first-match wins, specific rules must come before more general rules in the client configuration file. Signing Failed for ... Agent Refused Operation As mentioned, incorrect permissions on client files or directories are a common cause of authentication failure when attempting key-based authentication. Not all the errors are clear about that, however. Here is an example of a misleading error message which is actually caused by incorrect permissions: $ ssh -i ~/.ssh/key-ed25519 fred@server.example.org sign_and_send_pubkey: signing failed for ED25519 "/home/fred/.ssh/key-ed25519" from agent: agent refused operation fred@server.example.org: Permission denied (publickey). The solution for that is to ensure that no other accounts can read or write to the private key. No others should be able to write to the .ssh directory or its parent either. Debugging Chrooted SFTP Accounts With chrooted SFTP sessions, the most common problem seems to be bad directory permissions. The chroot directory, and all directories above it, must be owned by root and not writable by any other user or group. Even though these directories' group memberships do not have to be root, if any of them is not root then it must not be group writable either. Failure to use the correct ownership will result in not being able to log in with the affected accounts. The errors when login is attempted can look like this from the client side: $ sftp fred@192.0.2.206 fred@192.0.2.206's password: Connection to 192.0.2.206 closed by remote host. Connection closed. Older clients will look like this: $ sftp fred@192.0.2.206 fred@192.0.2.206's password: packet_write_wait: Connection to 192.0.2.206: Broken pipe Couldn't read packet: Connection reset by peer The error message is much clearer on the server side: Aug 4 23:52:38 server sshd[7075]: fatal: bad ownership or modes for chroot directory component "/home/fred/" Check the directory permissions for the chroot target and all directories above it. If even one is off it must be fixed so that it is owned by root and not writable by any others. There are many, many routes to get there. Here are two was to set up chroot permissions: One quick way to fix the permissions is to change both ownership and group membership of the directory to root. Same for all directories above the chroot target. $ ls -lhd /home/ /home/fred/ drwxr-xr-x 3 root root 4.0K Aug 4 20:47 /home/ drwxr-xr-x 8 root root 4.0K Aug 4 20:47 /home/fred/ That will work with the ChrootDirectory directive set to %h but has some drawbacks that will quickly become obvious when adding files or directories. Another easy way to fix the permissions is to change both the account's home directory and the ChrootDirectory directive. Arrange the account's home directory so that it is under a unique directory owned by root, such as the user name itself: $ ls -lhd /home/ /home/fred/ /home/fred/fred/ drwxr-xr-x 3 root root 4.0K Aug 4 20:47 /home/ drwxr-x--- 3 root fred 4.0K Aug 4 20:47 /home/fred/ drwxr-x--- 8 fred fred 4.0K Aug 4 20:47 /home/fred/fred/ Then chroot the account to the parent directory and combine that with an alternate starting directory working from the user name token with the -d option for the SFTP server. ChrootDirectory /home/%u ForceCommand internal-sftp -d %u That way, when the account connects it will see only its own directory and no other parts of the system. Debugging RC Scripts Interfering with SFTP Sessions Recent versions of the server do not launch either /etc/ssh/sshrc or ~/.ssh/rc when initiating an SFTP session. However, on older versions, these scripts will get executed for both SFTP and SSH, if present and the server is configured to run them. Therefore, in older versions one common mistake is when either /etc/ssh/sshrc or ~/.ssh/rc send anything at all to stdout instead of being quiet. The SFTP connection will drop when there are any extraneous data either direction on stdin, whether from the client or the server. The reason is that the output to stdout on the server is received by the client on stdin but matches no correct protocol and thus causes the client to disconnect with an error. However, even in the case of using the RC scripts, the response from the server must remain 8-bit clean or an error will occur: $ sftp server.example.org Received message too long 1400204832 That one message will be the main clue. Increasing the verbosity of the SFTP client with -v won't provide more relevant information. Upgrading sshd(8) might be a more practical option, when possible. When the RC scripts cause the SFTP client to disconnect, the standard server logs will show only that the client disconnected and not provide any information about why. At higher levels of logging, some extraneous reads and corresponding discards might be noticed but that is all. Below is a log sample recorded at the verbosity DEBUG3 showing such an example. ... debug2: subsystem request for sftp by user fred debug1: subsystem: exec() /usr/libexec/sftp-server Starting session: subsystem 'sftp' for fred from 198.51.100.38 port 37446 id 0 ... debug2: channel 0: read 13 from efd 12 debug3: channel 0: discard efd debug2: channel 0: read 12 from efd 12 debug3: channel 0: discard efd debug2: channel 0: read 15 from efd 12 debug3: channel 0: discard efd debug2: channel 0: read 18 from efd 12 debug3: channel 0: discard efd ... Again, new versions won't run the RC scripts when connecting for an SFTP session. On older versions, neither RC script may produce any output on stdout during use of SFTP or it will ruin the connection. If an RC script does produce output, it must be redirected to a system log, to a file, or sent to stderr instead of stdout. Regular interactive SSH connections are not disturbed by use of stdout and the client will just display whatever is sent. See the manual page for sshd(8) in the section "SSHRC" for more. The same restriction goes for any other part of the SSH service which runs over stdin and stdout, such as ProxyJump or some uses of ProxyCommand. So another example of potential interference would be when using LocalCommand with the client to specify a command to execute on the local machine after successfully connecting to the server. Any output from it also needs to be redirected to stderr. If LocalCommand ends up interfering with ProxyJump then the connection will appear to hang at the stage when stdout gets used. Debugging When An SSH Agent Has The Correct Private Key But Does Not Use It In older versions of OpenSSH, the public key must also be available on the client at the time the private key is loaded into the agent. If it is not then without the matching public key the agent will not be able to use that private key unless other arrangements are made. A symptom of this is although specifying the key via a run-time argument works, the same key does not work via the agent. Upgrading to a more recent version of OpenSSH is a better option. Otherwise a work-around is to specify the private key either as a run-time argument to the client or in the ssh_config(5) file, from there the client will find the correspondingly named public key file. Importantly the client will still use the key in the agent yet use the designated matching public key file, so the private key file does not have to contain anything at all and could even be empty. $ ssh -i some_key_ed25519 fred@server.example.org However, if it is undesirable to have the private key accessible on the file system or if the private key is only in the agent and not itself available via the file system, then the public key can be specified explicitly instead. $ ssh -i some_key_ed25519.pub fred@server.example.org Either way, another approach is to name the key in ssh_config(5) using the IdentityFile configuration directive. When a public key file is missing, it can easily be regenerated from the private key using ssh-keygen(1) with the -y option. SSH Client Error Messages And Common Causes As rehash of the above, below are some client-side error messages with some of the more common reasonshttps://ittavern.com/ssh-troubleshooting-guide/ SSH Troubleshooting Guide 2023-01-17 IT Tavern 2023-01-26 for those messages. Neither list of errors nor the reasons for the errors are comprehensive. The suggestions for causes just touch on one or two of the commonly seen reasons. There is no substitute for checking the actual logs, especially on the server. The server log files are usually /var/log/auth.log on most system or a variant of that name. Occasionally you will find the information in the file /var/log/secure instead. No address associated with name: $ ssh nonesuch.example.org ssh: Could not resolve hostname nonesuch.example.org: no address associated with name The destination's host name does not exist in DNS. Was it spelled correctly? Operation timed out: $ ssh 198.51.100.89 ssh: connect to host 198.51.100.89 port 22: Operation timed out There's not a system associated with that IP address, or else a packet filter is causing trouble. Connection timed out: $ ssh 198.51.100.89 ssh: connect to host 198.51.100.89 port 22: Connection timed out You can't get there from here. It is probable that the destination machine is disconnected from the network or that the network which it is on is not reachable. Connection refused: $ ssh www.example.org ssh: connect to host www.example.org port 22: Connection refused The destination system exists but there is no SSH service available at the port being tried. Is the destination right? It might be the wrong system, SSH might not be running at all, SSH might be listening on another port, or a packet filter might be blocking connections. Permission denied: $ ssh www.example.org fred@www.example.org: Permission denied (publickey,keyboard-interactive). That is usually a sign of a problem caused by the wrong user name, wrong authentication method, wrong SSH key or SSH certificate, or wrong file permissions in the authorized keys file on the destination system. If it is a matter of SSH key-based authentication, see the chapter on Public Key Authentication for more thorough coverage. If it is a question of SSH certificates, see the corresponding Certificate-based Authentication chapter. It can also be a matter of server-side settings with AllowGroups, DenyGroups, or similar configuration directives at the destination. Any of those possibilities can really only be identified and solved by checking the log output from sshd(8). No matching host key type found: $ ssh www.example.org Unable to negotiate with 198.51.100.89 port 22: no matching host key type found. Their offer: ssh-rsa,ssh-dss The SSH daemon on that server is really outdated. Contact the system administrator about an upgrade and get them moving. More details can be seen with the -v option on the client. $ ssh -v fred@example.org ... debug1: SSH2_MSG_KEXINIT received debug1: kex: algorithm: diffie-hellman-group-exchange-sha256 debug1: kex: host key algorithm: (no match) Unable to negotiate with 198.51.100.89 port 22: no matching host key type found. Their offer: ssh-rsa,ssh-dss In the above shell session, it is the server which is badly in need of an update as shown by the deprecated key exchange algorithms which it tries to use. Again, for emphasis, the solution is to update the outdated software. If a specific, old version of an particular operating system absolutely must be used for a while longer, see about getting a back port of the SSH daemon. Many GNU/Linux distros even have specific back port repositories. If you would like to see which key exchange algorithms the client supports try using the -Q option. $ ssh -Q kex diffie-hellman-group1-sha1 diffie-hellman-group14-sha1 diffie-hellman-group14-sha256 diffie-hellman-group16-sha512 diffie-hellman-group18-sha512 diffie-hellman-group-exchange-sha1 diffie-hellman-group-exchange-sha256 ecdh-sha2-nistp256 ecdh-sha2-nistp384 ecdh-sha2-nistp521 mlkem768nistp256-sha256 curve25519-sha256 curve25519-sha256@libssh.org sntrup761x25519-sha512 sntrup761x25519-sha512@openssh.com mlkem768x25519-sha256 mini 100%