Introducing Kubernetes Vault Webhook

Initially, we had the DevOps framework in which Development and Operation team collaborated to create an “Agile” ecosystem. But nowadays a lot of people are talking about the “DevSecOps” realm in which people do not treat security as an afterthought instead of that people are inculcating security in their development and operation practices.

Since we are a technology-enabled company, we also integrate the security best practices in the work we are doing, for example- secure infrastructure creation, application setup and scanning, WAF integration, etc. But nowadays people are quite fascinated about container-native technologies and they are shifting their microservices workload to different container orchestration technologies like ECS, Kubernetes, etc.

We are working with a lot of organizations that are using Kubernetes for managing their workloads and we are helping them to manage it in a very secure way. In my experience, a lot of people have asked me questions like — how can we manage secrets inside Kubernetes, I know most people will say Kubernetes has its own API object to manage the secret. But is it enough to manage the secrets in Kubernetes?

Let’s find out

Limitations with Kubernetes Secrets

• Rotation not supported — Secrets stored inside the Kubernetes cluster do not get automatically renewed or revoked, so all these things need to be done manually.
• Limits to Kubernetes — Secrets will be only limited to Kubernetes and its namespace i.e. it cannot be accessed outside the Kubernetes cluster.
• Base64 Encoding — Kubernetes uses base64 encoding to encrypt and decrypt data that can be easily decoded.
• Secrets not Encrypted — Kubernetes stores all secrets inside the ETCD database which is not encrypted by default, so anyone who has access to the ETCD server or API’s can read the data.
• Secrets Readable inside Pod — If anyone has access to get inside the pod, they will be able to read the value of the secrets through the environment variable.

Kubernetes Vault Webhook

As we are open-source enthusiasts people, we decided that we will try to create a solution around this. Then we started evaluating the best solution for this kind of problem and we came across a very interesting topic of Kubernetes i.e. Mutation Webhooks. So people who are new to webhook concepts let me try to put it in simple words, a webhook is an endpoint that generally works on Layer7 protocol with different REST API methods like PUT, POST, GET, etc. and only get triggered by an event.

Similarly, a Kubernetes mutation webhook is a webhook that listens for some special events from Kubernetes, and once the event is captured it mutates or changes the resource which we are deploying inside the Kubernetes cluster, for example:- Pods, Secrets, etc.

By using this amazing concept of mutation webhooks, we decided to create a solution called “K8s Vault Webhook” which can integrate with different secret managers to fetch the secrets from it and then inject it inside the Pods process environment. These are some supported features of K8s Vault Webhook:-

• K8s Vault Webhook listens for some specific Kubernetes events like pods.
• With some specific annotations, it can mutate the pod to inject secrets directly inside the pod’s process environment, so if you are able to get inside the pod as well still you will not be able to read the secrets from the environment variable.
• Supports multiple secret managers like- AWS Secret Manager, Azure Key Vault, Hashicorp Vault, and GCP Secret Manager.

The general architecture of K8s Vault Webhook looks like:-

For more information about this webhook, please check out the documentation link.

Conclusion

So in this blog, we have seen the limitations of the Kubernetes secret resource and how we tried to come up with a solution to overcome those limitations by creating a custom webhook inside Kubernetes.

In the next part of this blog, we will discuss that how can we use this webhook with Hashicorp Vault inside the Kubernetes cluster.
If you guys have any other ideas or suggestions around the approach, please comment in the comment section. Thanks for reading, I’d really appreciate your suggestions and feedback.

References:-
https://github.com/OT-CONTAINER-KIT/k8s-vault-webhook
https://github.com/innovia/secrets-consumer-webhook
https://i.pinimg.com/originals/98/68/e8/9868e87dc36fa2bffb357d708d457f31.gif