A production CloudFormation stack shows drift - resources have been modified outside of CloudFormation. How do you detect, investigate, and remediate drift without causing downtime?
Quick Answer
Use `aws cloudformation detect-stack-drift` to identify drifted resources, `describe-stack-resource-drifts` to see exact property changes, then remediate by either updating the template to match actual state or re-deploying to restore desired state, depending on whether the drift was intentional.
Detailed Answer
Drift Detection Mechanics
CloudFormation drift detection compares the actual resource configuration (from AWS API calls) against the expected configuration (from the last successful stack operation). It reports three statuses: IN_SYNC, MODIFIED (properties changed), and DELETED (resource removed). Not all resource types support drift detection - check the AWS documentation for supported types. Run detection via CLI, console, or schedule it via EventBridge + Lambda for continuous monitoring.
Investigation Workflow
When drift is detected, examine each drifted resource's property differences. The describe-stack-resource-drifts API returns the expected value, actual value, and difference type (ADD, REMOVE, NOT_EQUAL) for each property. Common drift sources: security teams adding tags, auto-scaling modifying instance counts, manual hotfixes during incidents, and AWS service-linked role modifications. Categorize drift as intentional (authorized changes) or unintentional (configuration drift).
Remediation Strategies
For unintentional drift (someone manually changed a security group): update the stack to reassert the desired state. Create a change set to verify what will happen, then execute it. For intentional drift (security team added a required tag): update the template to include the change, then deploy to bring the template in sync with reality. For complex drift (resource was replaced): you may need to import the new resource into CloudFormation using resource import.
Prevention
Implement preventive controls: SCP policies that deny direct resource modification for CloudFormation-managed resources (using aws:cloudformation:stack-id condition key), AWS Config rules that alert on changes to tagged resources, and IAM policies that restrict write access to production resources. Establish a culture where all changes go through IaC, even during incidents (use fast-track change sets).