Design a secrets management strategy using HashiCorp Vault for a Kubernetes-based microservices platform with 200 services. How do you handle secret rotation without downtime?
Quick Answer
Use Vault's Kubernetes auth method with per-service policies, dynamic secrets for databases, the Vault Agent sidecar for transparent secret injection, and dual-credential rotation for zero-downtime.
Detailed Answer
Architecture
Authentication
- Vault Kubernetes auth method: pods authenticate using their ServiceAccount JWT token - Each service gets a Vault role mapped to its ServiceAccount, with policies scoped to its secrets only - No static tokens or passwords stored anywhere
Secret Types & Handling
1. Dynamic database credentials (preferred): - Vault generates unique DB credentials per pod with TTL (e.g., 1 hour) - Credentials auto-expire — no rotation needed, no credential sharing between services - If compromised, blast radius is one pod's short-lived credential
2. Static secrets (API keys, third-party tokens): - Stored in Vault KV v2 (versioned) - Vault Agent sidecar or CSI driver injects secrets as files into the pod - Vault Agent watches for changes and hot-reloads when secrets rotate
3. PKI certificates: - Vault PKI secrets engine issues short-lived TLS certificates (24h TTL) - Auto-renewal via Vault Agent - mTLS between all services
Zero-Downtime Rotation (Dual-Credential Pattern)
1. Generate new credential (version N+1) in Vault 2. Both old (N) and new (N+1) credentials are valid simultaneously 3. Applications pick up new credential via Vault Agent file watch 4. After all pods have rotated (verify via audit log), revoke version N 5. Grace period ensures no request fails during the transition
Governance
- Vault audit logging to detect unauthorized access attempts - Sentinel policies to enforce naming conventions and TTL limits - Regular access reviews via Vault token accessor listing
Code Example
# Vault Kubernetes auth role (Terraform)
resource "vault_kubernetes_auth_backend_role" "order_service" {
backend = vault_auth_backend.kubernetes.path
role_name = "order-service"
bound_service_account_names = ["order-service"]
bound_service_account_namespaces = ["orders"]
token_policies = ["order-service-policy"]
token_ttl = 3600
}
# Dynamic DB credentials policy
path "database/creds/order-service-db" {
capabilities = ["read"]
}
path "secret/data/orders/*" {
capabilities = ["read"]
}
# Vault Agent sidecar annotation in pod spec
metadata:
annotations:
vault.hashicorp.com/agent-inject: "true"
vault.hashicorp.com/role: "order-service"
vault.hashicorp.com/agent-inject-secret-db: "database/creds/order-service-db"
vault.hashicorp.com/agent-inject-template-db: |
{{- with secret "database/creds/order-service-db" -}}
postgresql://{{ .Data.username }}:{{ .Data.password }}@db:5432/orders
{{- end }}Interview Tip
Lead with dynamic secrets as the primary strategy — they eliminate rotation entirely. For static secrets, explain the dual-credential rotation pattern. Mention audit logging and Sentinel policies to show you think about governance, not just plumbing.