Your team manages 15 microservices across dev, staging, and production environments, each with different configuration values, secrets, and feature flags. How do you use Pulumi ESC (Environments, Secrets, Configuration) to manage environment-specific configuration at scale?
Quick Answer
Pulumi ESC provides hierarchical environment definitions that compose and inherit from each other. Define base configurations, layer environment-specific overrides, integrate with external secret stores (Vault, AWS Secrets Manager), and consume them in Pulumi stacks, CLI tools, and application code through `esc run`.
Detailed Answer
What is Pulumi ESC
Pulumi ESC (Environments, Secrets, Configuration) is a centralized configuration management service separate from Pulumi IaC. It defines 'environments' as YAML documents containing key-value pairs, secrets, and references to external secret stores. Environments can import from other environments, creating a composition hierarchy. Unlike traditional .env files, ESC provides versioning, RBAC, audit logging, and integration with multiple secret backends.
Hierarchical Environment Design
Design your environments in layers: a base environment with defaults shared across all services, a base/aws-<region> with region-specific AWS credentials, a <service>/base with service-specific config, and <service>/<env> for environment-specific overrides. The composition order determines precedence - later imports override earlier ones. This reduces duplication dramatically: database connection strings, API URLs, and feature flags are defined once in the appropriate layer.
Secret Store Integration
ESC integrates natively with AWS Secrets Manager, HashiCorp Vault, Azure Key Vault, GCP Secret Manager, and 1Password. Instead of copying secrets into ESC, you define references: fn::secret::aws-secrets-manager: { secretId: 'prod/db-password' }. ESC resolves these at runtime, and secrets never leave the source store. This satisfies compliance requirements that mandate secret stores as the source of truth.
Consumption Patterns
ESC values can be consumed in three ways: (1) in Pulumi IaC stacks via pulumi config env add <env-name> which injects ESC values as stack config, (2) from the CLI via esc run <env> -- <command> which sets environment variables for any process, and (3) via the ESC SDK/API for programmatic access. This means ESC is not limited to Pulumi deployments - it can configure Terraform runs, Docker containers, CI/CD pipelines, and local development environments.