How does Vault inject secrets into Kubernetes pods (Agent vs. CSI)?

Think of two ways to get mail delivered to your apartment. The Vault Agent Injector is like a personal assistant who lives with you (sidecar container), picks up your mail from the post office (Vault), sorts it for you (renders templates), and keeps checking for new mail throughout the day (auto-renewal). The Vault CSI Provider is like a mail slot in your door (volume mount) where the building's mailroom staff (DaemonSet) delivers letters from the post office. Both get mail to you, but the assistant can do more while the mail slot is simpler and cheaper.

The Vault Agent Injector works through a Kubernetes mutating admission webhook. When a pod is created with certain annotations (vault.hashicorp.com/agent-inject: "true"), the webhook intercepts the pod creation and adds Vault Agent sidecar containers. An init container runs first to authenticate with Vault and fetch initial secrets before the app starts, so secrets are ready from the beginning. The sidecar container keeps running alongside the application, automatically renewing tokens and re-fetching secrets when they change or are about to expire. Secrets are written as files in a shared in-memory volume (tmpfs) mounted at /vault/secrets, keeping sensitive data off disk.

Under the hood, the Agent Injector runs as a Kubernetes Deployment in the vault namespace with a MutatingWebhookConfiguration that intercepts CREATE and UPDATE operations on pods. When triggered, it adds two containers: an init container that runs vault agent with -exit-after-auth for the initial secret fetch, and a sidecar that runs vault agent in daemon mode with auto-auth set up for Kubernetes authentication. The agent uses consul-template-style templates to format secrets however you need (plain text, JSON, env file, etc.). The CSI Provider takes a different path. It runs as a DaemonSet that implements the CSI (Container Storage Interface) provider spec. When a pod mounts a SecretProviderClass volume, the kubelet calls the CSI driver, which talks to the Vault CSI provider through a Unix socket to authenticate and fetch secrets, then mounts them as files in the pod.

In production, the choice between the two depends on your needs. Agent Injector is more popular because it handles secret rotation without restarting pods, supports template rendering for custom formats, and works with any Kubernetes distribution. But each pod gets a sidecar using about 25-75 MB of memory, which adds up across hundreds of pods. CSI Provider is more efficient since one DaemonSet serves all pods on a node, but it requires installing the Secrets Store CSI Driver separately and does not natively support automatic secret rotation without enabling the rotation feature flag. Many teams use both: Agent Injector for pods that need dynamic database credentials with auto-renewal, and CSI Provider for simpler cases like TLS certificates or static API keys.

The biggest gotcha with the Agent Injector is annotation sprawl and init container race conditions. If the init container fails to fetch secrets because Vault is briefly unavailable, the app container will start without secrets, which can cause crashes. Teams need proper readiness probes and retry logic. With CSI Provider, the key pitfall is that secrets are only fetched at pod startup by default. If a secret rotates in Vault, existing pods keep the stale value until they restart, unless you enable the rotation reconciler that polls Vault at set intervals. Another common mistake is not setting the right service account for the pod, which causes auth failures that show up as confusing mount errors instead of clear permission-denied messages.