How do Vault Agent and CSI Provider inject secrets into Kubernetes pods without code changes?

Think of two ways to get mail delivered to your apartment. A personal assistant (Vault Agent) picks up your mail, sorts it, puts specific letters in specific folders, and checks for new mail all day. A building mailroom slot (CSI Provider) delivers mail when you first open your mailbox but does not actively refresh it. Both save you from going to the post office yourself (hardcoding Vault API calls in your app), but they differ in how they handle ongoing updates.

The core problem both solve is getting secrets to applications without changing application code. Without them, every app needs Vault SDK integration, authentication logic, lease renewal code, and error handling for Vault being down. Many legacy or third-party applications cannot be modified at all. Both Vault Agent and CSI Provider authenticate to Vault on behalf of the application, fetch the requested secrets, and make them available as ordinary files the application can read like any config file.

Vault Agent works by injecting an init container that authenticates to Vault using Kubernetes auth (validating the pod's service account JWT), fetches the initial secrets, and writes them to a shared emptyDir volume using Go templates. Then the sidecar container keeps running to handle lease renewal and re-rendering when secrets change. The Agent template system supports custom formatting, so secrets can be written as .env files, JSON, YAML, or whatever the application expects. When a dynamic secret's lease is about to expire, Agent automatically renews it or fetches new credentials and updates the file. The Vault Agent Injector automates all of this through a mutating webhook that reads pod annotations.

The CSI Provider takes a different route. The Secrets Store CSI Driver creates a CSI volume that mounts secrets as files. The Vault CSI Provider plugin authenticates to Vault and fetches the specified secrets during the volume mount phase. Secrets show up as files in the mounted path. But since no sidecar keeps running, it does not handle lease renewal or dynamic secret rotation by default. The enableSecretRotation feature polls Vault at intervals, but the application needs to watch for file changes or pods need to restart for updated secrets to take effect.

The hidden cost of Vault Agent is resource overhead at scale. Each pod with the Agent sidecar uses 50-100 MB of memory and some CPU. In a cluster with 2,000 pods, that is 100-200 GB of memory just for secret injection sidecars. The gotcha with CSI Provider is the rotation gap: if a database credential is revoked in Vault and the CSI Provider has not refreshed yet, the application uses stale credentials and gets authentication errors. Pick Agent when dynamic secret rotation matters, and CSI Provider when simplicity and lower resource usage are the priority.