HashiCorp Vault became the go-to for enterprise secrets management with its 1.0 release in 2019. After three years of use in production environments, certain patterns have emerged as effective, while others have not.
The standout feature of Vault is dynamic secrets. These are credentials generated on demand that expire after a set period. For instance, a database secret engine can create a unique username and password for each service requesting credentials. These credentials are revocable and auditable, and the service never stores a long-lived credential. If a credential is compromised, it automatically expires, solving the rotation issues that plague static credentials.
One of the challenges with dynamic secrets is integrating them with existing systems. For example, we had to integrate Vault with a legacy database system that only supported a limited set of authentication mechanisms. It took several weeks to develop a custom plugin to support this integration. The plugin had to handle the complexities of the database's authentication protocol, which included a challenge-response mechanism. This experience taught us to carefully evaluate the feasibility of dynamic secrets for each use case.
Vault's PKI secrets engine acts as a certificate authority, issuing short-lived TLS certificates programmatically. In a microservices architecture, this means Vault PKI can provide certificates valid for hours, not years. When integrated with cert-manager for Kubernetes, Vault PKI can automatically issue and rotate certificates for Kubernetes workloads.
To give you an idea of the scale, we have seen Vault deployments with over 1000 nodes, handling tens of thousands of certificate requests per minute. This requires a highly performant storage backend, such as Raft with a distributed storage system like Amazon EBS or Google Persistent Disk. In one deployment, we saw a 30% increase in performance after switching to a more efficient storage backend.
Vault's Kubernetes auth method allows workloads in Kubernetes to authenticate using their Kubernetes service account token. This approach eliminates the need for pre-provisioned credentials. Instead, the workload's Kubernetes identity is used to obtain a Vault token, which then retrieves secrets. This is the recommended pattern for Kubernetes-based workloads, ensuring that secrets are not stored in environment variables or mounted files but are retrieved from Vault at runtime.
However, running Vault in high availability requires an integrated storage backend, such as Raft, and a strategy for unsealing. Using auto-unseal with cloud KMS, like AWS KMS or Azure Key Vault, removes the need for manual unsealing after restarts. The initial setup of Vault, including mount points, policies, auth methods, and the first set of secrets, is a significant undertaking that can take several days. Ongoing maintenance involves policy updates, engine upgrades, and audit log rotation. While Vault is powerful, it is not straightforward to manage.