How do you handle Terraform state file corruption in production? Describe your recovery process and how you prevent it from happening.
Quick Answer
Enable state file versioning in the backend (S3 versioning), use state locking (DynamoDB), and recover by restoring the previous state version. Prevent by never running terraform apply concurrently.
Detailed Answer
State Corruption Scenarios
1. Concurrent terraform apply without state locking 2. Manual edits to state file 3. Interrupted apply (network loss, process kill) leaving partial state 4. Provider bugs writing malformed state entries
Recovery Process
1. Stop all Terraform operations — prevent further corruption 2. Identify the corruption: terraform plan will usually show the error. Common: Error: Invalid state file 3. Restore from backup: - S3 backend: Use S3 versioning to restore previous state version - aws s3api list-object-versions --bucket my-tf-state --prefix env/prod/terraform.tfstate - Download the last known good version and upload as current 4. For partial corruption: Use terraform state rm to remove the corrupted resource, then terraform import to re-import it 5. Validate: Run terraform plan and verify the diff matches expected state
Prevention
- Always use remote backend with state locking (S3 + DynamoDB, GCS, Terraform Cloud) - Enable S3 versioning on the state bucket with lifecycle rules (keep 30+ versions) - CI/CD only: Never run terraform from local machines in production - State backup before destructive operations: terraform state pull > backup.tfstate - Use -lock-timeout=5m to handle transient lock conflicts
Production at Scale
Split state files by environment and component (networking, compute, data). Smaller state files reduce blast radius and lock contention. Use Terraform Cloud/Enterprise workspaces for state management at scale.
Code Example
# List state versions in S3 aws s3api list-object-versions \ --bucket my-tf-state \ --prefix prod/terraform.tfstate \ --max-items 5 # Restore a previous version aws s3api get-object \ --bucket my-tf-state \ --key prod/terraform.tfstate \ --version-id "abc123" \ restored.tfstate # Remove corrupted resource from state terraform state rm 'aws_instance.corrupted' # Re-import the resource terraform import 'aws_instance.corrupted' i-0123456789abcdef0 # Verify terraform plan
Interview Tip
This question tests operational maturity. Always mention prevention first (locking, versioning, CI-only), then recovery. Interviewers want to see you've dealt with this in production, not just read about it.