How do you manage secrets in Kubernetes using HashiCorp Vault with auto-rotation and zero application changes?

Imagine a bank where every employee carries a permanent master key that opens every door. If one key is stolen, every room is compromised, and changing the locks requires visiting every employee. Now imagine a system where employees receive temporary badges that expire every hour, open only the doors they need, and are automatically replaced before expiration. If a badge is stolen, it becomes useless within minutes, and the security team knows exactly who accessed what and when. HashiCorp Vault brings this temporary-badge model to Kubernetes secrets management.

Traditional Kubernetes Secrets are base64-encoded, stored in etcd, and remain static until someone manually updates them. This creates several problems in regulated banking environments. Secrets do not rotate automatically, so a compromised credential remains valid indefinitely until discovered. Secrets are accessible to anyone with RBAC read permission on the namespace, creating a broad attack surface. There is no centralized audit log showing which pod accessed which secret and when, making SOX compliance evidence difficult to produce. Vault solves these problems by becoming the single source of truth for all secrets, generating credentials dynamically, and maintaining a complete audit trail.

The Vault Agent Injector is the most common integration pattern for Kubernetes. It works through a mutating admission webhook that intercepts pod creation requests. When a pod has Vault annotations, the webhook injects a Vault Agent sidecar container. This sidecar authenticates to Vault using the pod's Kubernetes ServiceAccount token, retrieves the requested secrets, writes them to a shared in-memory volume as files, and continuously renews or re-fetches them before their lease expires. The application reads database credentials from a file path like /vault/secrets/db-creds instead of from environment variables or Kubernetes Secrets. Because the application already reads configuration from files in most frameworks, this requires zero code changes. The CSI Provider is an alternative that mounts secrets as a CSI volume, useful for teams that prefer volume mounts over sidecar injection.

Dynamic secrets are Vault's most powerful feature for banking environments. Instead of storing a static database password, Vault generates a unique database username and password for each pod with a configurable time-to-live. When the lease expires, Vault automatically revokes the credential at the database level. This means each pod has its own credential that can be traced in audit logs, compromised credentials expire automatically, and credential rotation requires no deployment or restart. For AWS resources, Vault's AWS secrets engine generates temporary IAM credentials with specific policies. For TLS certificates, Vault's PKI engine issues short-lived certificates that rotate without manual intervention.

The production gotcha that experienced DevSecOps engineers watch for is lease management under scale. When hundreds of pods each hold dynamic database credentials, the total connection count at the database can exceed pool limits during deployments or scaling events. Teams must configure max_ttl and default_ttl carefully, implement connection pooling in the application, and monitor Vault's lease count as a capacity metric. Another critical consideration is Vault's own availability: if the Vault cluster is unreachable during a pod startup, the sidecar cannot inject secrets and the pod fails to start. Production Vault deployments use multi-replica Raft storage, cross-availability-zone distribution, and auto-unseal with AWS KMS to maintain five-nines availability. Banking teams also configure Vault audit devices to stream every secret access event to Splunk or the enterprise SIEM, creating the tamper-evident audit trail that OCC examiners require.