How do you handle Terraform state corruption and recovery?

Terraform state corruption is one of the most dangerous operational incidents because the state file is the single source of truth mapping your configuration to real infrastructure. Think of it like a hospital's patient registry: if the registry is corrupted, you do not know which patient is in which room, and any action you take risks harming the wrong patient. State corruption can cause Terraform to destroy resources it thinks do not exist, create duplicates of resources it cannot find, or fail entirely with cryptic deserialization errors.

State corruption occurs in several ways. The most common is a partial write during terraform apply: if the process is killed mid-apply (OOM killer, network interruption, CI timeout), the state file may contain a partially updated resource. Another common cause is manual state file editing with JSON syntax errors. Concurrent writes without proper locking can interleave state updates. Provider bugs can write malformed attribute values. And rarely, S3 eventual consistency (in older S3 behavior) could serve a stale state version.

The first line of defense is prevention. Enable S3 versioning on your state bucket so every state write creates a new version, giving you point-in-time recovery. Enable MFA delete on the bucket to prevent accidental or malicious version deletion. Use DynamoDB state locking to prevent concurrent writes. Run terraform plan before apply to catch state inconsistencies early. Implement CI/CD pipelines that prevent direct state manipulation.

When corruption occurs, follow a structured recovery procedure. First, immediately stop all Terraform operations on the affected workspace. If your CI/CD pipeline has queued runs, cancel them. This prevents compounding the corruption.

Second, assess the damage. Run terraform state pull to download the current state file and inspect it. Check the JSON structure: is it valid JSON? Are the serial number and lineage fields present? Use jq to examine specific resources. Compare the state against your actual infrastructure using AWS CLI or console.

Third, attempt recovery from backup. If S3 versioning is enabled, list previous versions with aws s3api list-object-versions, download a known-good version, and push it back using terraform state push. The state push command validates the state format and updates the serial number. Be careful with the -force flag: it skips lineage checking, which is a safety mechanism that prevents pushing state from the wrong workspace.

Fourth, if no backup is available, perform selective state surgery. Use terraform state rm to remove the corrupted resource entries, then terraform import to re-import the existing infrastructure resources. This is tedious for large configurations but preserves the non-corrupted portions of state. For each imported resource, run terraform plan to verify the import produced a state entry that matches your configuration.

Fifth, as a last resort for total state loss, you can rebuild the entire state from scratch using terraform import for every resource. This is the infrastructure equivalent of a bare-metal restore. Tools like terraformer and former2 can help by scanning your AWS account and generating import commands, but they require careful validation.

After recovery, conduct a post-mortem. Implement additional safeguards: S3 bucket replication to a separate account for disaster recovery, automated state backup jobs that copy state to a different storage system, monitoring on state file size and serial number for anomaly detection, and regular recovery drills where you practice restoring state from backup in a non-production workspace.