How do Vault secrets engines work (KV, Database, AWS, PKI)?

Think of secrets engines as specialized departments in a bank. The KV engine is the safe deposit box department where you store and retrieve items. The Database engine is a temporary badge office that prints access cards expiring after a shift. The AWS engine is a valet service creating temporary parking passes. The PKI engine is a notary office issuing certified documents. Each department has its own procedures, but they all follow the same bank's security policies.

Secrets engines in Vault are components that store, generate, or encrypt data. You enable them at specific paths (like secret/, database/, aws/, pki/) and each path is completely isolated from the others. The KV (Key-Value) engine comes in two versions: v1 offers simple storage, while v2 adds versioning, soft delete, and metadata tracking. The Database engine connects to real databases like PostgreSQL, MySQL, or MongoDB and creates unique credentials per request, each with a configurable time-to-live. The AWS engine generates IAM users, STS tokens, or assumed-role credentials on the fly. The PKI engine works as a full certificate authority, issuing X.509 certificates with parameters you define.

Under the hood, each secrets engine works like a virtual filesystem. When you request credentials from database/creds/payments-readonly, Vault routes the request to the Database engine, which connects to the configured database, creates a new user with the right permissions, and hands back the credentials along with a lease ID. This lease ID is important because Vault tracks all active leases and automatically revokes credentials when the lease expires. For a database credential, revocation means running a DROP USER or similar statement against the database. For PKI, requesting a certificate from pki/issue/web-server triggers certificate generation using the configured CA, and the certificate's validity period becomes its lease. Each engine keeps its own configuration, connections, and state independently.

In production, organizations typically mount multiple instances of the same engine for different environments and teams. For example, you might have database/prod/ and database/staging/ as separate mounts with different connection strings and policies. The Database engine is especially powerful for microservices because each pod gets unique credentials. If a pod is compromised, you revoke only that specific lease without touching other services. The AWS engine is invaluable for CI/CD pipelines where build jobs need temporary AWS access without storing long-lived IAM keys. The PKI engine eliminates the headache of managing certificates by hand, automating issuance, renewal, and revocation, often cutting certificate provisioning time from days to milliseconds.

The biggest mistake with secrets engines is getting lease TTLs wrong. Setting database credential TTLs too short causes connection storms as hundreds of pods simultaneously request new credentials. Setting them too long defeats the whole point of dynamic secrets. A common PKI mistake is exposing the root CA through Vault instead of keeping it offline and only mounting an intermediate CA. For KV v2, teams often forget that deleting a secret only soft-deletes it, and the data stays recoverable until you explicitly destroy the version. This matters for compliance when handling sensitive data like personally identifiable information.