Most advances in DevOps today are being leveraged for speed. Continuous integration and continuous delivery (CI/CD) is driven by the need to meet internal demands, meet customer demands, and adapt to the competition. However, as the headlines constantly remind us, security can’t be overlooked in our rush to deploy.
No matter where your organization is on the spectrum from monolithic code to microservices, or annual to continuous releases, security is often considered an impediment to progress. Of course, the ideal situation is for tools to deliver speed and security.
Why native security is not sufficient
Starting with applications, there’s often a need for one application to access another application or database either within or outside the enterprise. This is how secrets (passwords, keys) end up hard-coded, or barely better, in configuration files, into the applications themselves. We’ve seen many instances of hard-coded credentials posted publicly in Github, and we’re sure this happens in internal repositories as well. DevOps tools often provide their own method of secrets management. Kubernetes provides a method to store secrets, which I’ll discuss later. Jenkins, Chef, and Puppet all have some type of vault. So why not use them?
Secrets rotation is far easier from one vault location as opposed to coordinating several locations in a variety of different tools
A single centralized vault provides efficiency and security. Many of your applications and DevOps tools need access to the same assets, so having one vault enables the creation of a unified policy for access to those assets. Secrets rotation is far easier from one vault location as opposed to coordinating several locations in a variety of different tools. A centralized vault also enables simplified auditing and alerting from that single source rather than having to pull logs from various tools.
It’s important to note that none of this is very easy, if possible at all, with hard-coded credentials in applications or secrets spread over several DevOps tools. For both applications and DevOps tools, when secrets are needed, an API call to the centralized vault is ideal.
Kubernetes security with DevOps Secrets Vault
As mentioned, Kubernetes provides a mechanism for the applications in pods to access secrets. They are managed in the etcd (pronounced et-cee-dee) distributed database along with all the other cluster configuration information.
Beyond the non-centralized vault issues mentioned above, there are two known issues with this method. The first is that any secrets stored in etcd are available to any node in the cluster. So an intrusion into any node in the cluster allows access to all the other nodes via lateral movement. The other issue is the secrets are stored in plain text (base64 encoded) in the etcd database.
The DevOps Secrets Vault Kubernetes Plug-in uses a different approach. It consists of a Broker in its own pod and a Client that acts as a sidecar within the pod of any application that needs a Secret. Let’s look at how this technique works. (See diagram)