You need to manage Pulumi state for a platform team supporting 50+ microservices across dev, staging, and production. How do you design the state backend architecture, and what happens when state gets corrupted?
Quick Answer
Use Pulumi Cloud or a self-hosted S3 backend with per-stack state files, organize stacks as <project>/<environment>, implement state locking, and maintain backup/export procedures for corruption recovery using `pulumi stack export/import`.
Detailed Answer
Backend Selection
Pulumi supports multiple state backends: Pulumi Cloud (SaaS), self-hosted Pulumi Cloud (Enterprise), S3-compatible storage, Azure Blob, Google Cloud Storage, or local filesystem. For enterprise use, Pulumi Cloud provides built-in encryption, audit logging, RBAC, state history, and concurrent operation locking. If you must self-host, use S3 with versioning enabled, server-side encryption (SSE-KMS), and DynamoDB for state locking (configured via pulumi login s3://bucket-name).
Stack Organization
Organize stacks using the pattern <org>/<project>/<stack>. For 50+ microservices, create a project per service with stacks per environment: acme/order-service/dev, acme/order-service/prod. Shared infrastructure (VPC, DNS, databases) gets its own project with stack references for cross-stack outputs. This isolation ensures a bad deployment in one service cannot corrupt another service's state.
State Corruption Recovery
State corruption typically happens when a deployment is interrupted (SIGKILL, network failure) or when someone manually modifies cloud resources outside Pulumi. Recovery workflow: First, export the corrupted state with pulumi stack export > state.json. Examine the JSON to identify the corruption (orphaned resources, incorrect outputs, duplicate URNs). Edit the JSON to fix issues, then import with pulumi stack import < state.json. For resources that exist in the cloud but not in state, use pulumi import to bring them under management. For resources in state but deleted from cloud, remove them from the exported JSON.
Operational Safeguards
Enable S3 bucket versioning so every state change is recoverable. Run pulumi stack export as a pre-deployment step in CI/CD and archive state snapshots. Use Pulumi Cloud's built-in state history to compare state before and after deployments. Implement stack policies that prevent deletion of critical stacks without explicit confirmation.