How does Vault generate dynamic database credentials, and what happens at scale?

Think of a hotel that issues temporary room keys instead of permanent ones. Each guest gets a unique key that expires at checkout. If the guest wants to stay longer, they visit the front desk to extend it. If every guest checks in at 3 PM at the same time, the front desk becomes a bottleneck. Vault's dynamic secrets work the same way: each application instance gets unique database credentials that expire, but mass simultaneous requests stress the system.

Dynamic secrets exist because static, shared database passwords are a security risk. When every microservice shares one password and it leaks, every service is compromised and rotating the credential means coordinating all consumers at once. Vault's database secrets engine creates a unique username and password (or certificate) per request, with each one tracked by a lease. The lease has a TTL (time-to-live) and a max TTL. Services can renew the lease before it expires to keep using the same credentials, or let it expire and request fresh ones.

Under the hood, when a service reads from the creds endpoint (e.g., vault read database/creds/payments-readonly), Vault connects to the target database using a pre-configured admin connection, runs the creation SQL defined in the role (CREATE USER with specific privileges), generates a random password, stores a lease record in its storage backend, and returns the credentials with the lease ID and TTL. When the lease expires or gets revoked, Vault runs the revocation SQL (DROP USER or ALTER USER) to remove access. A background process in Vault scans for expired leases and revokes them in batches.

At production scale with 500 pods rolling out at once, each requesting database credentials, Vault faces three bottlenecks: Vault's own request handling capacity, the database's ability to handle rapid CREATE USER statements, and the connection limit on the database. If each of 500 pods gets unique credentials and opens a connection pool of 10, that is 5,000 database connections, which can blow past PostgreSQL's max_connections. You need to tune max_open_connections on the Vault database config to limit how many credential-creation connections Vault opens at once, set reasonable default_ttl and max_ttl to balance security with renewal overhead, and put connection poolers like PgBouncer between the application and database.

The sneaky gotcha is lease revocation storms. When Vault's lease tracker falls behind due to high load or a brief outage, expired leases pile up. When the tracker catches up, it tries to mass-revoke, sending hundreds of DROP USER statements to the database at once. This can spike database CPU and cause lock contention, hurting application queries. Watch the vault.expire.num_leases metric, set appropriate max_open_connections, and stagger pod rollouts to avoid credential request thundering herds.