Over the past few years, I've watched Kubernetes configuration and secret management evolve from a simple set of ConfigMaps and Secrets into a complex and rich ecosystem of external secret stores, GitOps-compatible encrypted secrets, and dynamic secret injection.
One of the earliest and still most useful patterns I've seen is the file system mount pattern for ConfigMaps. This allows configuration to be updated without restarting pods: the mounted ConfigMap files update when the ConfigMap changes, and applications that watch the file system for config changes can reload without restart. Spring Cloud Kubernetes and ASP.NET Core's FileSystemWatcher-based config reload are two examples that use this pattern.
I recall a project where we used Spring Cloud Kubernetes to manage database connection strings for a distributed e-commerce application. The ConfigMap was updated weekly with new connection string values, and the application reloaded the configuration without a restart. The benefits were clear: reduced downtime and easier maintenance of configuration.
Another pattern that's gained traction is the External Secrets Operator. This tool synchronises secrets from external secret stores like HashiCorp Vault, AWS Secrets Manager, Azure Key Vault, and GCP Secret Manager into Kubernetes Secrets. The application reads from Kubernetes Secrets without knowing the secret source, which gives it a nice level of abstraction. The ESO handles authentication, secret retrieval, and automatic rotation when the external secret changes.
For example, we used the External Secrets Operator to manage database credentials for a PostgreSQL database. The ESO synchronised the credentials from AWS Secrets Manager into Kubernetes Secrets, and the application used the Kubernetes Secrets for database connections. This decoupled the application from the secret source, making it easier to switch to a different secret store or add additional secret sources.
One of the biggest risks with ConfigMaps and Secrets in Kubernetes is that they're mutable by default. This means that a ConfigMap change takes effect immediately in mounted volumes, which can cause unexpected configuration updates in production. To mitigate this risk, I recommend using the immutable pattern: create new ConfigMap versions instead of editing existing ones, and update Deployment annotations to trigger a rolling update when the ConfigMap changes.
The benefits of immutability are clear: reduced risk of configuration drift and easier auditing of configuration changes. However, this approach also comes with a trade-off: increased storage requirements for multiple ConfigMap versions and additional complexity for updating Deployment annotations.
The downward API provides another way to manage dynamic configuration without requiring ConfigMap changes. By injecting pod metadata as environment variables, you can avoid the need to update ConfigMaps whenever your application's environment changes.
One of the biggest advantages of the downward API is its simplicity: no need to update ConfigMaps or use additional tools like the External Secrets Operator. However, it also has its limitations: only pod metadata can be injected as environment variables, which may not be sufficient for all use cases.
For GitOps workflows, Bitnami Sealed Secrets provides a way to encrypt Kubernetes Secrets and store them in Git. The SealedSecret custom resource contains the encrypted secret; only the Sealed Secrets controller can decrypt it. This allows all Kubernetes configuration, including secrets, to be stored in Git repositories without exposing secret values.