Your team accidentally ran 'terraform destroy' on a production environment. The state file shows all resources as destroyed, but the S3 backend has versioning enabled. Walk through the complete recovery process including state restoration and resource reconciliation.
Quick Answer
Restore the previous state file version from S3 versioning, verify it matches actual infrastructure (some resources may have been destroyed before you caught it), import any resources that were actually deleted and recreated, and implement safeguards to prevent recurrence.
Detailed Answer
Immediate Response (First 5 Minutes)
1. STOP any running Terraform operations immediately (kill the process if terraform destroy is still running) 2. Identify what was actually destroyed vs. what's still running (the destroy may have been partial) 3. Notify the incident team - this is a P1 production incident
State Recovery from S3 Versioning
S3 versioning keeps every version of the state file. The current version shows an empty state (post-destroy). The previous version contains the full resource inventory. Restore by downloading the previous version and pushing it back.
Reconciliation Process
After restoring state, run terraform plan. Three scenarios: 1. Resources still exist: Plan shows no changes (terraform destroy was interrupted early) 2. Resources were destroyed: Plan shows resources to be created (they need to be recreated) 3. Partial destruction: Mix of existing and missing resources
For destroyed resources: Run terraform apply to recreate them. For stateful resources (RDS, ElastiCache), check if automated backups exist and restore from the latest backup before running apply.
Prevention Measures
1. Enable prevent_destroy lifecycle on critical resources 2. Use Terraform Cloud/Spacelift with approval workflows for destroy operations 3. Implement CI/CD-only applies (no local terraform destroy allowed) 4. AWS-level protection: Enable RDS deletion protection, S3 bucket policies preventing deletion, CloudFormation stack termination protection for critical resources 5. State file locking with DynamoDB to prevent concurrent operations 6. Break-glass procedure documentation for destroy operations requiring multi-person approval