Kubernetes is growing in popularity and its use within organizations is maturing. According to the 2020 Cloud Native Computing Foundation (CNCF) annual survey, 83% of organizations were using Kubernetes in production, and 28% of organizations had more than 11 Kubernetes production clusters. At the same time, security concerns around Kubernetes are increasing. A Red Hat survey of DevOps professionals on Kubernetes adoption and security demonstrated that:
One important aspect of Kubernetes security is the use of TLS encryption to protect all traffic between clusters. Implementing mutual TLS (mTLS) authentication is a best practice for enhancing the security of Kubernetes and ensuring that only authenticated entities are communicating with your clusters."
To prevent security incidents from happening in the first place, Kubernetes documentation mentions that “TLS should be enabled for every component that supports it to prevent traffic sniffing, verify the identity of the server, and (for mutual TLS) verify the identity of the client.”
In the meantime, it is important to understand that TLS should be employed to encrypt all communications in the cluster between services, not just ingress. However, this security control is often overlooked considering that the cluster is secure and there is no need to provide encryption in transit within the cluster. (There are also many cases where mTLS is used, but the application platform team misconfigures mTLS or configures it out of compliance with the organization’s security policy. But that is a topic for another blog.)
OWASP Kubernetes Security cheat sheet mentions that “Kubernetes expects that all API communication in the cluster is encrypted by default with TLS.” However, there is a drawback with TLS: TLS guarantees authenticity but by default this only happens in one direction: the client authenticates the server, but the server doesn’t authenticate the client.
This is where mTLS comes useful—mTLS makes the authenticity symmetric.
Mutual authentication, also known as two-way authentication, is a security process in which entities authenticate each other before actual communication occurs. In a network environment, this requires that both the client and the server must provide digital certificates to prove their identities. In a mutual authentication process, a connection can occur only if the client and the server exchange, verify, and trust each other’s certificates.
The default option of TLS is to only authenticate in one direction because the client’s identity is often irrelevant—this is the case with websites, for example. Of course, not validating client identity makes sense for serving web pages, but there are plenty of types of communication where the identity of the client is important. API calls are one example: if you’re calling a service, then the service needs to know who you are for various reasons, from providing personalized information to sending you the bill. You can’t make an API call without providing some kind of client identity.
Many services use an authentication token to validate the identity of the clients. Instead of using this token, mutual authentication through mTLS provides some characteristics that are particularly beneficial for services like Kubernetes.
mTLS authentication is done entirely outside of the application without requiring any application level features for creating, registering, or managing identities. With mTLS, a new client can authenticate itself instantly. What is more, the application doesn’t need to know anything about authentication or provide endpoints to manage it.
These characteristics are what make mTLS so ideal for securing every communication within Kubernetes.
mTLS is a great way to secure all communications between microservices in a Kubernetes environment for the following reasons:
First, you secure all communications. When we implement our applications as multiple services, sensitive customer data is sent across the network between these services. Without proper encryption, anyone who gets access to the network can potentially read this sensitive data and falsify requests.
Second, it’s about auditing and authorization. When calls to services are coming from authenticated entities, you can audit and record appropriate metrics. Auditing gives you insights on what and how the client is doing, and you can also spot abnormal activities and prevent breaches. This is also important for authorization. Is the authenticated call entitled, with authorized to access the specific data? Authentication and authorization work together to ensure the confidentiality and integrity of your communications.
Third, it takes care of authentication. Since the authentication takes place outside of the app, your developers are not required to build flows and procedures for managing identity verification. Instead, they can focus on building and securing business logic and providing a well-functioning app.
Fourth, it is about user experience. Traditional token authentication can become cumbersome for returning users. Instead, mTLS authentication takes place once, at the platform level, and the user is not required to validate their identity when they jump between services.
mTLS is a great fit for securing the communication between microservices. But there’s a catch. Machine identity management faces the challenge of creating and distributing all these certificates. This certificate distribution challenge is compounded by the fact that in environments like Kubernetes, a “service” is in fact an ephemeral instance that can be created or destroyed on the fly, each of which needs its own set of certificates.
This challenge is further compounded by certificate rotation as a response to certificate compromise. With certificate lifecycle shortening every year, you need to re-issue the certificates before they expire. This means we need to repeat the whole CSR (certificate signing request) and certificate flow, for each ephemeral service, every now and then.
This is where cert-manager comes in handy. cert-manager, developed by the Jetstack team at Venafi, makes certificate authorities and certificates first-class resource types in the Kubernetes API, enabling developers to easily request machine identities for applications, whilst platform and security teams can maintain control and visibility. Learn more about the success of cert-manager here.
(This post has been updated. It was originally published on April 7, 2022.)