What is Vault's lease system, and what happens operationally if lease revocation falls behind at scale?
Quick Answer
Every dynamic secret Vault issues comes with a lease (a TTL and a renewal/revocation contract), and Vault tracks all outstanding leases so it can revoke the underlying credential (e.g. drop a database role) when the lease expires or is explicitly revoked. At scale, if Vault's lease count grows faster than leases are being revoked/expired (e.g. a bug causing excessive credential issuance, or revocation itself failing against a struggling backend), Vault's lease storage and revocation workload can grow large enough to degrade Vault's own performance and slow down new secret issuance.
Detailed Answer
Every response from a dynamic secrets engine includes lease metadata (lease_id, lease_duration, renewable) and Vault persists that lease in its own storage backend alongside a scheduled expiration. A background process continually checks for expired leases and calls the corresponding secrets engine's revocation logic (e.g. connecting to the database to DROP the specific role that was created for that lease). Clients can also proactively renew a lease before it expires if they still need the credential, up to the secret's configured max_ttl.
At scale, two related problems show up in real deployments: (1) lease/credential sprawl — if an application has a bug that requests a new dynamic credential far more often than necessary (e.g. per-request instead of once at startup, or a retry loop that doesn't reuse an existing lease), the sheer volume of outstanding leases and the corresponding load on both Vault's storage backend and the target system (e.g. a database straining under constant CREATE/DROP ROLE churn) can become a self-inflicted incident; (2) revocation backlog — if the backing system (database, cloud provider API) is itself degraded or rate-limiting Vault's revocation calls, expired leases can pile up faster than they're cleaned up, growing Vault's lease storage and potentially leaving stale credentials valid longer than intended if revocation keeps failing silently in the background.
Operationally, teams monitor vault_expire_num_leases (or equivalent) as a capacity signal, cap max_ttl aggressively on secrets engines to bound the blast radius of any credential issuance bug, and treat repeated revocation failures against a backend as a page-worthy signal rather than a silently-retried background task.
Interview Tip
The distinction between 'lease sprawl from over-issuance' and 'revocation backlog from a struggling backend' as two separate failure modes is what shows real operational depth here.