Secrets in Kubernetes: CSI Driver vs. ESO

In our previous blogs on Workload Identity, we explained how to securely authenticate to Azure services from AKS without using long-lived secrets; a best practice for Azure authentication. What about other secrets your apps might need, though? Think of API keys for external services, passwords for systems without Entra ID integration, or other sensitive configuration data. 

Standard Kubernetes Secret objects aren’t ideal for this, so it’s better to manage them centrally in an external vault like Azure Key Vault. How to make them available to your AKS pods in a secure and manageable way? Two popular tools come into play here: the Secrets Store CSI Driver, and the External Secrets Operator (ESO). Let’s compare them!

Why are Kubernetes Secrets falling short? 

Before we dive into the tools, let’s briefly review why standard Kubernetes Secret objects are problematic for sensitive data: 

• The data in a secret is only Base64 encoded by default, which means anyone with Kubernetes API access to the object can easily decode it. 
• The encoded data is stored in etcd, the Kubernetes database. While managed services like AKS enable encryption-at-rest on etcd, the secret remains readable via the API. In self-managed clusters, etcd encryption isn’t enabled by default, so this can easily be overlooked. 
• Obviously, storing Base64-encoded secrets directly in your Git repository is far from safe. Check out our blog on structuring Git repositories for more on that!

Secrets Store CSI Driver: Ultimate flexibility 

The Secrets Store CSI Driver is a specific implementation of the Container Storage Interface (CSI) standard for secrets. You install the driver on your cluster and configure a SecretProviderClass custom resource per application or namespace. This resource defines which secrets to retrieve from which external provider. In our case: Azure Key Vault.  

The CSI Driver offers two ways to make secrets available to your pod: 

As a volume mount 

Secrets are mounted as files in a (temporary, in-memory) volume, directly within the pod. The big advantage here is that no Kubernetes Secret object is ever created, so the sensitive data never enters the Kubernetes API or etcd. This is the most secure option, especially if you’re unsure about etcd encryption. The downside is that your application must be adapted to be able to read secrets from files. 

As a Kubernetes Secret 

You could also configure it to create or update a standard Kubernetes Secret object with the values from Key Vault. You do this by adding the secretObjects property to the SecretProviderClass custom resource. This is useful for apps that expect environment variables (which you can mount from this secret), but it does mean the (Base64 encoded) secret will exist in the API and (encrypted in AKS) in the etcd. 

In both cases, secrets only become available after the pod has started and the CSI Driver has completed its work. Applications that need their secrets immediately at startup may therefore run into issues. A common workaround is to use init containers that wait until the secrets are available before launching the main application. However, this does add some extra complexity. 

The driver can also periodically check for updates in Key Vault using enableSecretRotation. If you’re using volume mounts, the files will be updated, but your application must be able to reload them. Syncing to a Kubernetes secret, that object will be updated accordingly.

External Secrets Operator (ESO): Simple and readily available 

The External Secrets Operator (ESO) takes a different approach. It’s a Kubernetes Operator that continuously monitors specific custom resources: 

• SecretStore (or ClusterSecretStore): This is where you define the connection to your external secret manager, such as Azure Key Vault. It’s crucial that you also configure the authentication method here, preferably Workload ID for Azure Key Vault. 
• ExternalSecret: In this resource, you specify which secrets from the configured SecretStore should be synced and to which Kubernetes Secret object they should be written. 
• Using the refreshInterval, you can control how frequently this sync occurs. Keep in mind that you may need to restart the pod to see the changes, depending on how you mount environment variables. 

ESO always syncs secrets to a standard Kubernetes Secret object; either by creating a new one or by updating an existing one. Unlike the CSI Driver, there’s no second option. That’s why it’s so important to have etcd encryption at rest enabled (although this is standard in AKS). 

Because the operator manages and syncs the Secret object independently of your pods, the secret is already available when the pod starts. Your application can immediately use environment variables mounted from this ESO-managed Kubernetes secret. That is also why we tend to use ESO more often in our AKS environments. 

Sealed Secrets: A third option? 

Another tool you might come across is Sealed Secrets. It works differently: you encrypt your Kubernetes Secret manifest locally, using a public key. The result (a SealedSecret object) can be safely committed to Git. A controller in the cluster with the private key can then decrypt it back into a standard Secret. 

Sealed Secrets solves the issue of storing secrets in Git but doesn’t integrate with external managers like Key Vault for centralised management and rotation. It’s a viable option if you don’t use an external vault, but it’s less powerful than CSI/ESO in that scenario. In other words: we don’t generally recommend it. 

Workload ID: A must! 

Positively essential for both CSI Driver and ESO: to communicate safely with Azure Key Vault, we strongly recommend using Workload ID. It makes sure the sync tool itself doesn’t need any long-lived credentials to access Key Vault. 

It links the Service Account of the CSI Driver pod or the ESO Operator pod to an Azure Managed Identity with the appropriate permissions on Key Vault. This way, the entire chain – from external store to pod – works without secrets. You can find all the details in our two-part blog on Workload ID. 

How do you choose the right tool? 

For secrets that you can’t avoid using Workload ID, standard Kubernetes Secret objects are too risky and unsuitable for GitOps workflows. The best approach is to manage everything centrally in Azure Key Vault and sync it to your cluster, using the right tool. 

Both the Secrets Store CSI Driver and the External Secrets Operator (ESO) provide strong solutions, ideally combined with Workload ID for authentication to Key Vault. 

The CSI Driver offers maximum security with its volume mount option (no Kubernetes Secrets involved) but may be more complex due to application compatibility and startup timing issues. ESO is often easier to use and resolves the startup problem but always stores the secret as a(n encrypted in AKS) Kubernetes Secret. 

So, your choice depends on your balance between security, application requirements, and ease of use. At CloudFuel, we typically use ESO, especially in managed AKS environments. 

Need help choosing or implementing your secret management approach? Get in touch, we’re happy to help you.