Secret Management on Kubernetes with Vault
Secret management has always been a hot topic since the rise of Kubernetes. Many cloud providers have sold us their Vault implementations, and then some of us may have also heard of the open source Vault by HashiCorp.
Personally, for me, Vault implementations have always seemed somewhat over complicated. Why is that ?
How would I attempt to secure secrets on Kubernetes?
In an environment like Kubernetes I can use the built-in secret abstraction that the API provides. It allows me to decouple secrets so developers can refer to secret by name and Kubernetes will mount the actual secret into the applications file system at the time of scheduling. Furthermore, I can lock secret objects down with built-in RBAC facilities that Kubernetes provides.
That means we can prevent the reading of secrets by people. We can also lock it down even more by given the application some identity, using the service account API, and then scope read of certain secrets to only certain applications. This can further help limit the blast radius incase of a control plane service account token breach. Kubernetes also provides encryption at rest in ETCD if the API server is configured with something like the --encryption-provider-config
So why do we even need Vault ?
If Kubernetes provides abstraction, access control, encryption all out the box, then why do we need a Vault ? It took me a while to understand this until I digged in to take a look at what HashiCorp Vault provides.
So what is wrong with the my Kubernetes way? Well, nothing really, but there are still a couple of drawbacks:
- A human is still responsible for adding the secret to Kubernetes
This is where the most flaws exist with secret management implementations. How many of you have production TLS, API tokens, keys, database credentials and other keys on your laptop right now ? How many of you have emailed, sent it via chat, and distributed these to other team members because they needed it ? How many of you commit these files to source control ? A quick search on Google and I find many security breaches involve credentials being leaked and also caused mostly by human error.
2. Most secrets live for years
How many of you actually put a TTL on each of your secrets ? How often are you expiring TLS certificates, rotating database credentials and API keys ? Most people never do. As long as it involves people and manual operations, it is almost never done. Most of us have secrets in Kubernetes that we never replace. Kubernetes may provide a platform for us to build controllers that can force rotate these, however it’s not the responsibility of Kubernetes to do so.
3. Vaults require encryption keys
I always understood that a Vault requires encryption keys, so I would have to store one key, and trade it for another like the actual secret. So questions I always had was how do I now protect the Vault key ? If I store the Vault key in Kubernetes and run workloads at a large scale, it sounds like a security nightmare to me. We’ll fortunately, HashiCorp Vault does things differently.
How does Vault solve these problems ?
Vault is an identity translator. Instead of us storing encryption or access keys of a Vault, we pass Vault something else that we know and trust. Vault then trades it for the actual secret we need.
In the world of Kubernetes we already have the service account API and we should be running all applications under these service account objects in RBAC enabled clusters with minimal to no privileges. These are all best practises that I highly encourage when using Kubernetes. Many attack vectors in Kubernetes include attackers using remote code execution on insecure web servers to obtain Kubernetes tokens for API access. You should always protect the control plane at all costs. Your applications or pods should not access the Kubernetes API under normal circumstances.
Vault runs with all this in mind. It uses a less privileged service account, a low privileged Linux user account in the container and uses RBAC policies which you can fine tune further. So let’s go!
- No encryption keys in Kubernetes
Vault uses the Shamir Secret Sharing algorithm to generate it’s encryption key dynamically. That means no encryption key on disk or in Kubernetes.
But how ?
The algorithm involves an unseal process where each part of a key is supplied until the Vault can generate it’s encryption key. When Vault initializes for the first time it spits out 5 key parts. 3 of them are required to manually unseal the Vault. In High Availability mode, at least one Vault instance needs to be unsealed. If Kubernetes reschedules an instance, it would lose its encryption key and start sealed whereby it would need to be unsealed again. This can be automated using Vaults auto unseal mechanism.
The idea here is that no single person should ever be able to unseal the Vault.
2. Dynamic secret injection
One of the features that got me hooked on Vault was its ability to dynamically generate secrets and inject them into our applications file system. Now this may sound scary to DB admins or security engineers, but you need to dig deeper to understand how this all works, so don’t panic!
An application on a Vault enabled Kubernetes platform is unaware that Vault even exists. It has no mechanisms to request a secret. Everything we’ll talk about happens at the control plane level. That means even if an application was breached, it has no means to access vault or the dynamic secret injection features. But how?
When Kubernetes schedules an application, Vault uses a MutatingWebhookConfiguration to inject a sidecar process into the application’s pod. The sidecar process grabs the service account object that the application pod is running as and passes it to Vault. Vault verifies the identity of this application with the Kubernetes API. Hereby we pass the service account as something we trust, and Vault will validate and verify the secret we have access to. Vault also checks policies for this service account to ensure it can access the secrets requested in the applications deployment annotations. If all is well, Vault will pass the secret to the sidecar process which will place it on the application’s file system.
3. Secret life cycle management
With Dynamic secret injection, Vault removes the requirement for humans to place secrets manually into an environment as I explained above. In addition to all these features, Vault also manages the full lifecycle of the secret. Based on policies, it can revoke and create new secrets allowing us to automatically expire and rotate credentials, tokens, SSH keys and certificates. It will handle this automatically and provision secrets each time an application instance starts up. When the application deschedules, the secret is revoked and deleted from the system.
One takeaway here is that Vault and Kubernetes makes a really great pair since they’re both designed to handle these types of dynamic workloads.
Where to from here ?
I’ve got many requests from folks over on my Youtube channel to explore secret management on Kubernetes. I am super keen to see this all in action and to see the boundaries we can push with Hashicorp Vault.
In my secret management series I will explore all these features, starting with an Introduction guide where we spin up a Vault, discuss all the concepts of Vault to ensure you understand how to operate it in production. We’ll then take each video to the next step, enabling end-to-end TLS encryption, managing static secrets, and finally dynamic secrets.
Let’s go!
