How do you detect and manage infrastructure drift in Terraform?

Think of a restaurant menu board. The manager updates the board (Terraform code), but a waiter changes the daily special price by hand on the physical sign (console change). The menu board and the sign now disagree. Drift detection is the manager checking the sign against the board every morning, and remediation is deciding whether to update the board or fix the sign.

Infrastructure drift occurs when real-world resources diverge from Terraform's state file. This happens when someone makes changes through the AWS console, another automation tool modifies resources, or AWS auto-applies changes like security patches. Terraform does not continuously monitor infrastructure — it only detects drift when you run terraform plan, which triggers a refresh that reads current resource attributes from cloud APIs.

terraform plan -refresh-only is the detection tool. It reads the current state of every resource in the state file and shows what has changed without proposing any modifications. The output shows 'Objects have changed outside of Terraform' with specific attribute differences. For automated detection, run this command on a schedule (daily via CI) and alert when the exit code indicates drift (terraform plan -refresh-only -detailed-exitcode returns exit code 2 when drift exists). AWS Config rules provide complementary detection for specific compliance violations like open security groups.

Remediation depends on intent. If the manual change was a valid emergency fix (someone tightened a security group during an incident), update the HCL code to include the change and apply. If the change was unauthorized or accidental, run terraform apply with the existing HCL to revert the resource to its declared state. For resources manually created outside Terraform, use terraform import to bring them under management, then write the corresponding HCL.

The non-obvious gotcha is that running terraform apply -refresh-only updates the state file to match reality, which makes Terraform think the manual change is now the desired state. If you later modify the HCL and apply, the manual change persists because it is now the state baseline. Always treat refresh-only as a detection command, not a fix. Also, some drift is expected and harmless — AWS tags added by auto-scaling, EKS-managed security group rules, or Lambda layer versions updated by deployment pipelines. Use lifecycle ignore_changes for attributes managed by external systems.