Let’s be honest, following corporate security policy is probably NOT top of mind for developers. DevOps teams are focused on rapid SLAs, continuous delivery and compressed timelines. So, when they need a certificate to test code, they are going to take the easiest route. In many cases, they will simply follow the script within their DevOps platform. While those scripts are likely to provide everything your developers need to issue certificates, they don’t necessarily have your overall security as a top priority.
Before we tackle the DevOps challenge, here’s a refresher on the overall reasons why you have a public key infrastructure (PKI) in the first place. Internal keys and certificates let you know which machines and applications you can trust in your network. That’s why you spend so much time managing your trust store and developing relationships with certificate authorities (CAs) that you can trust.
This is where DevOps can throw a wrench in that trust model. In the interest of speed and agility, developers often circumvent the policies that you have developed to keep your network secure and functional. By doing this, they create a false sense of trust in unauthorized certificates, whether they are issuing self-signed certificates or spinning up their own internal CAs that are outside of corporate security policy.
In defense of developers, they never claimed to be PKI experts. So, they simply may not realize the implications of using self-signed certificates or the potential hazards of spinning up their own internal CAs. It’s just that there is no easy way for them to request certificates that comply with corporate policy. First, they need to know where the internal CA is, then they must be granted access to it and possess the proper credential to authenticate. Once they pass those hurdles, they need to know how to request a certificate with the proper characteristics, including who’s using it, what it will be used for and how long it should be valid.
Because this process for requesting a compliant certificate can be rather lengthy, it’s no surprise that developers look for an easier way. DevOps platforms provide it by making it easy for developers to issue unauthorized certificates with a script or instructions that allow them to spin up their own internal CA. They can then issue as many certificates as they need as fast as they need them.
The underlying assumption is that these “test” certificates are to be used only in development would be replaced by corporate-compliant certificates when the application moves to production. But this is not always the case. So, organizations are exposed to a veritable avalanche of certificates that they can’t see or they may not even know about.
We’ve seen how this can play out with many companies who turned to us for help. At one site, a network scan revealed 140 unauthorized internal CAs. That’s 140 different private keys that may or may not be stored with adequate security. Basically, this organization had 140 CA keys floating around that could potentially be compromised and used to sign illegitimate certificates that would then be trusted by the organization.
But unauthorized CAs raise another issue. Every time developers spin up a CA, they are distributing that CA as a root of trust within the organization. Accordingly, they create a set of applications that are configured to “trust” this CA and the certificates that it issues. Without the proper visibility into your application trust stores, it’s very difficult to remediate this issue. Even if you re-issue these certificates under corporate policy, you’re still left with the challenge of weeding out the unauthorized CAs from your trust store.
We recently unearthed an even more alarming worst practice for unauthorized CAs. Using TrustNet, Venafi’s global certificate discovery and reputation service, we found 3,904 certificates at one site with the exact same subject DN and serial number. It’s likely that these were created by developers who followed the default script to spin up an internal CA and request their own unauthorized certificates. The obvious challenge is that it’s almost impossible to determine which of these 3,904 certificates should be trusted.
So, you’re faced with a dilemma. Do you force your DevOps teams to follow security policies for certificates and jeopardize their SLA? Or do you allow them to create self-signed certificates to move as fast as possible and boost your population of unauthorized internal CAs? The answer probably is neither. Ultimately, you need to migrate your DevOps to a safer process that gives you full visibility into all certificates, while at the same time providing developers with easy access from within DevOps frameworks.
At Venafi, that’s our vision. And we’re working hard to make it a reality.