Although Secure Shell (SSH) is the most broadly used security protocol for remotely managing Unix/Linux, routers, firewalls, and other systems, most organizations have limited or no formal SSH policies or management in place. Many security practitioners and managers outside of the Unix teams have only a cursory knowledge of SSH—many may not be able to tell you whether SSH uses certificates or public keys, let alone how broadly it is used or the risks it poses if not properly managed. In this blog series, I aim to provide information that will help you gauge whether your SSH implementation provides the security it is intended to, including a base understanding of what SSH is, how it works (at a high level), SSH risks, and best practices for reducing those risks.
SSH is a replacement for Telnet, a protocol that was designed to enable remote command line operations but transmitted passwords and other data across the network in the clear. System administrators now use SSH to perform remote administration for nearly all systems with the exception of Windows and mainframes—and there are even options for using SSH with Windows and mainframes. SSH is also broadly used by administrators to automate operations (scripting) and for file transfers between systems.
Unlike other security tools, SSH is generally not centrally managed. Instead, SSH is most often managed by individual administrators for the servers they control. Consequently, most organizations do not have a central view or way of controlling the configuration of SSH or the access it provides. This is quite astonishing, considering the level of privileged access SSH enables for many mission critical systems. Because of this lack of central oversight, administrators are left to their own devices (pun intended) and this results in significant security risks.
To understand the inherent risks of SSH, it first helps to better understand its basic operation. Instead of attempting to write a description of the major components of SSH and how it works at a high level, I’ve created a video.
As you can see from the video, even with only three players (Sally and two servers) you end up with a lot of SSH keys to track. In most enterprises, the result is a web of untracked trust relationships (created by authorized keys that provide persistent access), which can look a lot like the following:
As mentioned earlier, most organizations have no inventory of these SSH trust relationships. Keep in mind that SSH is used for privileged access (i.e., root or sudo). Without an inventory of SSH keys, many IT organizations have had a policy of not changing (rotating) SSH keys because they are wary inadvertently increasing the risk of outages. This concern is largely due to the fact that SSH works through tightly coupled private and public keys. If you update a private key but don’t update all the corresponding authorized (public) keys, you may cause a critical automated process that launches in the middle of the night to stop working. It only takes one of those outages, and the corresponding management meltdown, for IT organizations to decide that they NEVER change or touch SSH keys. Unlike certificates, SSH keys don’t have an expiration date, so decisions like this don’t consider the unintended security consequences that quietly accumulate, unnoticed and unaddressed.
So now you end up with a bunch of SSH keys that have been in place for years. Even when employees or applications are terminated, everyone is afraid to remove or update the SSH keys they had access to because they don’t know what might happen when they do.
Hmmm. Can you see how all of this might result in potential backdoor keys that provide unintended access, either for terminated employees or attackers who’ve been able to strategically place keys because there is no oversight or inventory?