Terraform Production Issues

Symptom

All Terraform operations across every pipeline return 'Error acquiring the state lock' and no deployments can proceed. The lock has been held for over 4 hours with no active Terraform process running.

Error Message

Error: Error acquiring the state lock

Error message: ConditionalCheckFailedException: The conditional request failed
Lock Info:
  ID:        a1b2c3d4-e5f6-7890-abcd-ef1234567890
  Path:      prod-terraform-state-us-east-1/infrastructure/production/terraform.tfstate
  Operation: OperationTypeApply
  Who:       runner@ip-10-0-47-218
  Version:   1.6.4
  Created:   2026-06-18 02:14:33.827145 +0000 UTC
  Info:      

Terraform acquires a state lock to protect against two processes
writing to state simultaneously. The lock has been held for 4h12m
and may have been left behind by a crashed process.

You may force-unlock this state lock by running:
  terraform force-unlock a1b2c3d4-e5f6-7890-abcd-ef1234567890

Root Cause

A GitLab CI runner executing a terraform apply against the production workspace was terminated mid-operation when the underlying EC2 spot instance was reclaimed by AWS. The runner process was killed with SIGKILL, which prevented Terraform's normal graceful shutdown and lock release sequence from executing. The DynamoDB lock record in the terraform-state-locks table retained the lock entry with the ID a1b2c3d4-e5f6-7890-abcd-ef1234567890, and because the process that held it no longer existed, no process could release it. All subsequent Terraform operations including plan, apply, and even state list commands for the production workspace were blocked because they all require acquiring the state lock. The CI pipeline had no timeout or dead-lock detection mechanism, and the on-call engineer was not alerted because the monitoring only checked for failed applies, not for lock age. This resulted in a 4-hour outage window during which no infrastructure changes could be deployed to production.

Diagnosis Steps

1. 1.aws dynamodb scan --table-name terraform-state-locks --region us-east-1 --output json | jq '.Items[] | select(.LockID.S | contains("production"))'
2. 2.aws dynamodb get-item --table-name terraform-state-locks --key '{"LockID": {"S": "prod-terraform-state-us-east-1/infrastructure/production/terraform.tfstate"}}' --region us-east-1
3. 3.kubectl logs -n gitlab-runner deployment/gitlab-runner --since=6h | grep -i 'terraform\|spot\|interrupt\|sigkill'
4. 4.aws ec2 describe-spot-instance-requests --filters 'Name=instance-id,Values=i-0a1b2c3d4e5f67890' --region us-east-1 --query 'SpotInstanceRequests[].Status'
5. 5.terraform force-unlock -dry-run a1b2c3d4-e5f6-7890-abcd-ef1234567890

Solution

The immediate resolution required force-unlocking the stale DynamoDB state lock. First, the team verified that no active Terraform process was actually running by checking all CI runners and SSH sessions. Then they executed terraform force-unlock a1b2c3d4-e5f6-7890-abcd-ef1234567890 which removed the orphaned lock record from DynamoDB. After the unlock, they ran terraform plan to verify the state was consistent and no partial writes had corrupted it. A follow-up terraform apply completed the partially-applied changes from the crashed run. To prevent recurrence, the team implemented a Lambda function triggered by CloudWatch Events that monitors DynamoDB lock age and automatically alerts via PagerDuty when any lock exceeds 30 minutes. They also migrated CI runners from spot instances to on-demand instances for production pipelines and added SIGTERM handling with graceful shutdown timeouts.

Commands

terraform force-unlock a1b2c3d4-e5f6-7890-abcd-ef1234567890

terraform plan -out=recovery.tfplan 2>&1 | tee /var/log/terraform/recovery-plan.log

terraform apply recovery.tfplan

aws dynamodb scan --table-name terraform-state-locks --region us-east-1 --output table

Symptom

Production RDS database, ElastiCache cluster, and 14 EC2 instances were destroyed during what was supposed to be a staging environment teardown. Customer-facing services went down immediately and data loss was confirmed.

Error Message

aws_db_instance.production_primary: Destroying... [id=prod-primary-db-us-east-1]
aws_db_instance.production_primary: Still destroying... [4m10s elapsed]
aws_db_instance.production_primary: Still destroying... [8m20s elapsed]
aws_db_instance.production_primary: Destruction complete after 12m33s
aws_elasticache_replication_group.prod_redis: Destroying... [id=prod-redis-cluster]
aws_instance.app_servers[0]: Destroying... [id=i-0abc123def456789a]
aws_instance.app_servers[1]: Destroying... [id=i-0abc123def456789b]
...
aws_instance.app_servers[13]: Destroying... [id=i-0abc123def456789n]

Destroy complete! Resources: 47 destroyed.

Error: Failed to delete S3 bucket prod-assets-cdn-us-east-1: BucketNotEmpty: The bucket you tried to delete is not empty
  status code: 409, request id: A1B2C3D4E5F6G7H8

Root Cause

An engineer intended to run terraform destroy against the staging workspace to tear down a temporary load-testing environment. The engineer's terminal session was connected to the production workspace from an earlier debugging session, and the workspace context was not verified before executing the destroy command. The Terraform configuration used shared variable files with workspace-conditional logic, so the plan output showed resource names that looked plausible for either environment since both used similar naming conventions with only a prefix difference. The engineer approved the destroy plan without carefully reading the resource identifiers, which clearly showed 'prod-' prefixed names. The CI/CD pipeline had no workspace-aware guardrails or mandatory approval gates for destroy operations in production workspaces. Additionally, the production workspace lacked lifecycle prevent_destroy meta-arguments on critical resources like the RDS instance, ElastiCache cluster, and the primary S3 buckets. There was no Sentinel or OPA policy preventing destroy operations against production-tagged resources during business hours.

Diagnosis Steps

1. 1.terraform workspace show
2. 2.aws rds describe-db-instances --db-instance-identifier prod-primary-db-us-east-1 --region us-east-1 2>&1
3. 3.aws rds describe-db-snapshots --db-instance-identifier prod-primary-db-us-east-1 --snapshot-type automated --query 'DBSnapshots | sort_by(@, &SnapshotCreateTime) | [-1]' --region us-east-1
4. 4.aws cloudtrail lookup-events --lookup-attributes AttributeKey=EventName,AttributeValue=DeleteDBInstance --start-time 2026-06-18T02:00:00Z --end-time 2026-06-18T04:00:00Z --region us-east-1
5. 5.terraform state list 2>&1 | head -50
6. 6.cat terraform.tfstate.backup | jq '.resources | length'

Solution

Immediate recovery focused on restoring the production database from the most recent automated RDS snapshot taken 35 minutes before the destroy event. The team executed aws rds restore-db-instance-from-db-snapshot to recreate the primary database, then reconfigured security groups and parameter groups to match the original. For the EC2 instances, the team ran terraform apply against the production workspace with the original variable files, which recreated all 14 app server instances from the latest AMI. The ElastiCache cluster was recreated similarly via Terraform apply. After infrastructure was restored, the team used terraform import to bring all recovered resources back under state management where automatic recreation had not handled it. Data loss was limited to approximately 35 minutes of database transactions which were partially recovered from application-level write-ahead logs. The total recovery took 6 hours. Post-incident, the team added lifecycle prevent_destroy blocks to all critical resources, implemented OPA policies that reject destroy plans in production, and added mandatory Slack approval workflows for any destroy operations.

Commands

aws rds restore-db-instance-from-db-snapshot --db-instance-identifier prod-primary-db-us-east-1 --db-snapshot-identifier rds:prod-primary-db-us-east-1-2026-06-18-01-30 --db-instance-class db.r6g.2xlarge --region us-east-1

terraform workspace select production && terraform plan -out=restore.tfplan

terraform apply restore.tfplan

terraform import aws_db_instance.production_primary prod-primary-db-us-east-1

Symptom

Terraform state shows duplicate resource entries, missing attributes, and lineage mismatch errors. Plans show resources that already exist as needing creation, and existing resources are not recognized.

Error Message

Error: Error refreshing state: state snapshot was created by Terraform v1.6.4,
which is newer than current v1.6.3; upgrade to Terraform v1.6.4 or later to work
with this state.

Additionally, the state file contains resources that reference providers
that are no longer available in the configuration:
  - registry.terraform.io/hashicorp/aws (2 resources orphaned)

Error: Saved plan is stale

The current state has changed since the plan was created. The plan is
no longer valid. Please create a new plan.

State serial: 4287 (expected: 4285)
State lineage: "d8f2a1b3-c4e5-6f78-9012-abcdef345678"

Root Cause

Two CI pipelines triggered simultaneously when two pull requests were merged within seconds of each other. Both pipelines acquired the DynamoDB state lock sequentially but operated on different versions of the state. The first pipeline completed its apply and wrote state serial 4286, but the second pipeline had already read state serial 4285 during its plan phase and cached it locally. When the second pipeline attempted to apply, it detected the serial mismatch and failed partway through, but not before writing partial resource changes to the state. The S3 state backend's versioning captured both writes, but the resulting state file contained a mixture of resource definitions from both applies. Some resources were duplicated with conflicting attribute values, while others lost their dependency metadata. The root issue was that the CI system did not enforce sequential pipeline execution for the same Terraform workspace, and the DynamoDB lock granularity did not cover the full plan-then-apply cycle as a single atomic operation. The plan was generated before the lock was acquired, creating a time-of-check to time-of-use race condition.

Diagnosis Steps

1. 1.aws s3api list-object-versions --bucket prod-terraform-state-us-east-1 --prefix infrastructure/production/terraform.tfstate --max-items 10 --region us-east-1
2. 2.aws s3api get-object --bucket prod-terraform-state-us-east-1 --key infrastructure/production/terraform.tfstate --version-id v1_prev_version /tmp/state_before.json --region us-east-1 && cat /tmp/state_before.json | jq '.serial, .lineage'
3. 3.terraform state list 2>&1 | sort | uniq -d
4. 4.terraform state pull | jq '.resources | group_by(.type + "." + .name) | map(select(length > 1)) | .[].[] | .type + "." + .name'
5. 5.grep -c 'pipeline' /var/log/ci/terraform-*.log | sort -t: -k2 -rn | head -5

Solution

Recovery required restoring the state from the last known good version in S3. The team used aws s3api get-object with the --version-id flag to download the state file from before the corruption (serial 4285). They then disabled all CI pipelines to prevent further writes. Using terraform state push -force, they overwrote the corrupted state with the clean version. Next, they ran terraform plan to identify the delta between the restored state and actual infrastructure. Resources created by the first successful apply were imported back into state using terraform import for each resource. Resources from the failed partial apply that were actually provisioned in AWS were either imported or manually destroyed depending on whether they were needed. After reconciling the state with reality, a clean terraform plan showed zero changes, confirming full recovery. The team then implemented a CI pipeline mutex using a dedicated DynamoDB-based semaphore that covers the entire plan-and-apply cycle as one atomic unit, preventing concurrent Terraform runs against the same workspace.

Commands

aws s3api get-object --bucket prod-terraform-state-us-east-1 --key infrastructure/production/terraform.tfstate --version-id QUpbz3m4L9OHK7yZSnGh2kV_version_id /tmp/clean_state.json --region us-east-1

terraform state push -force /tmp/clean_state.json

terraform plan -out=reconcile.tfplan -refresh=true 2>&1 | tee /var/log/terraform/reconcile-plan.log

terraform import aws_instance.app_server[0] i-0abc123def456789a

Symptom

After upgrading the AWS provider from v4.67.0 to v5.31.0, terraform plan shows 200+ resources marked for destruction and recreation. Critical production resources including RDS instances, EKS clusters, and ALBs are all flagged for replacement.

Error Message

# aws_lb.production_alb must be replaced
-/+ resource "aws_lb" "production_alb" {
      ~ id                         = "arn:aws:elasticloadbalancing:us-east-1:123456789012:loadbalancer/app/prod-alb/50dc6c495c0c9188" -> (known after apply)
      ~ subnets                    = [
          - "subnet-0a1b2c3d4e5f67890",
          - "subnet-0b2c3d4e5f6789012",
        ] -> (known after apply) # forces replacement
      ~ name                       = "prod-alb" -> (known after apply)
        # (12 unchanged attributes hidden)
      - enable_http2               = true -> null # forces replacement

      ~ subnet_mapping {
          # This attribute changed from a set to a list in provider v5.x
          ~ subnet_id = "subnet-0a1b2c3d4e5f67890" -> (known after apply)
        }
    }

Plan: 203 to add, 15 to change, 218 to destroy.

Root Cause

The AWS Terraform provider v5.x release included several breaking changes that affected resource schema definitions across multiple resource types. The most impactful changes were the conversion of subnet_mapping from a set type to a list type in aws_lb resources, the removal of the deprecated enable_http2 attribute in favor of a new protocol_version attribute in aws_lb_listener, changes to the internal state representation of aws_iam_role inline policies, and the renaming of several attributes in aws_eks_cluster from camelCase to snake_case. Because Terraform compares the state representation against the provider's schema, and the v5.x provider expected different attribute names and types, Terraform concluded that the existing resources could not be updated in place and needed to be destroyed and recreated. This affected 218 resources across the production infrastructure including 3 ALBs serving production traffic, 2 EKS clusters running 47 microservices, 12 RDS instances with a combined 8TB of data, and numerous supporting resources. The provider upgrade was applied directly in production without first testing in a staging environment or reviewing the v4-to-v5 migration guide published by HashiCorp.

Diagnosis Steps

1. 1.terraform plan -no-color 2>&1 | grep -c 'must be replaced'
2. 2.terraform plan -no-color 2>&1 | grep 'forces replacement' | sort | uniq -c | sort -rn
3. 3.terraform state show aws_lb.production_alb | grep -E 'subnet|enable_http2'
4. 4.diff <(terraform providers lock -platform=linux_amd64 2>&1) <(cat .terraform.lock.hcl) || true
5. 5.terraform version && cat .terraform.lock.hcl | grep -A2 'provider.*hashicorp/aws'

Solution

The team first rolled back the provider version constraint to ~> 4.67 in versions.tf and ran terraform init -upgrade to restore the working provider. With production stabilized, they began the migration in a staging environment. The migration required a multi-step approach: first, they wrote a Python script to parse the Terraform state file and identify all resources affected by the v5.x schema changes. For each affected resource type, they used terraform state mv to rename attributes where possible. For resources where the schema type changed (like the set-to-list conversion in aws_lb.subnet_mapping), they used terraform state rm followed by terraform import to re-import the resource under the new schema without destroying it. The script processed each resource type in batches, running terraform plan after each batch to verify zero-diff convergence. They also updated all HCL configuration files to use the new attribute names and types required by v5.x. The complete migration was tested three times in staging before executing in production during a maintenance window. The production migration was performed with terraform apply -target for each resource batch, taking 4 hours total with zero downtime.

Commands

sed -i 's/version = "~> 5.31"/version = "~> 4.67"/' versions.tf && terraform init -upgrade

terraform state rm aws_lb.production_alb && terraform import aws_lb.production_alb arn:aws:elasticloadbalancing:us-east-1:123456789012:loadbalancer/app/prod-alb/50dc6c495c0c9188

terraform plan -target=aws_lb.production_alb -out=alb-migration.tfplan

terraform state mv 'aws_eks_cluster.prod' 'aws_eks_cluster.prod' -state=terraform.tfstate -state-out=terraform.tfstate.migrated

Symptom

Terraform operations intermittently fail with lock acquisition timeouts during peak CI hours (9 AM - 11 AM UTC). Some pipelines succeed after retry but most fail, creating a backlog of 30+ queued deployments.

Error Message

Error: Error acquiring the state lock

Error message: RequestLimitExceeded: Provisioned throughput exceeded for table
'terraform-state-locks'. The level of configured provisioned throughput for
one or more global secondary indexes of the table was exceeded. Consider
increasing the provisioned throughput for the table or the indexes.

Lock Info:
  Path:      prod-terraform-state-us-east-1/infrastructure/terraform.tfstate

Terraform acquires a state lock to protect against two processes writing
to state simultaneously.

Retry limit (3) exceeded. The state lock could not be acquired within
the configured timeout of 60s.

aws_dynamodb_table: ProvisionedThroughputExceededException:
  Rate of requests exceeds the allowed throughput. Decrease the request
  rate or increase the table's provisioned capacity.
  Table: terraform-state-locks, RCU: 5/5, WCU: 5/5

Root Cause

The DynamoDB table terraform-state-locks was provisioned with 5 Read Capacity Units (RCU) and 5 Write Capacity Units (WCU), which was sufficient when the team had only a handful of Terraform workspaces and infrequent deployments. As the platform grew to 24 microservices across 6 environments, the number of concurrent Terraform operations during peak hours exceeded the provisioned capacity. Each Terraform operation performs at minimum 2 DynamoDB operations: a conditional PutItem to acquire the lock and a DeleteItem to release it. With 30+ pipelines competing during peak hours, the table received approximately 60+ write requests per minute, far exceeding the 5 WCU provisioned. DynamoDB throttled the excess requests, causing Terraform's lock acquisition to fail after the default 3 retry attempts within the 60-second timeout. The issue was exacerbated by the team's CI configuration, which triggered Terraform plan on every pull request update and apply on every merge, creating bursts of concurrent lock requests. Auto-scaling was not enabled on the table, and the team had not monitored DynamoDB consumed capacity metrics.

Diagnosis Steps

1. 1.aws dynamodb describe-table --table-name terraform-state-locks --region us-east-1 --query 'Table.ProvisionedThroughput'
2. 2.aws cloudwatch get-metric-statistics --namespace AWS/DynamoDB --metric-name ThrottledRequests --dimensions Name=TableName,Value=terraform-state-locks --start-time 2026-06-18T08:00:00Z --end-time 2026-06-18T12:00:00Z --period 300 --statistics Sum --region us-east-1
3. 3.aws cloudwatch get-metric-statistics --namespace AWS/DynamoDB --metric-name ConsumedWriteCapacityUnits --dimensions Name=TableName,Value=terraform-state-locks --start-time 2026-06-18T08:00:00Z --end-time 2026-06-18T12:00:00Z --period 300 --statistics Sum --region us-east-1
4. 4.aws dynamodb scan --table-name terraform-state-locks --region us-east-1 --select COUNT
5. 5.grep -r 'lock-timeout' /etc/terraform/*.tfrc 2>/dev/null || echo 'No lock timeout configured'

Solution

The immediate fix was to increase the DynamoDB provisioned throughput from 5/5 RCU/WCU to 25/25 using the AWS console to unblock queued pipelines. For the long-term fix, the team switched the table to on-demand (PAY_PER_REQUEST) billing mode, which automatically scales to handle any traffic pattern without manual capacity planning. They updated the Terraform backend configuration for the locks table itself using terraform apply against the infrastructure module that managed the DynamoDB table. Additionally, they increased the Terraform lock timeout from the default 0s to 300s using -lock-timeout=300s in all CI pipeline configurations, which allows Terraform to retry lock acquisition for 5 minutes instead of failing immediately. The team also staggered CI pipeline schedules so that not all microservice deployments triggered simultaneously and implemented a pipeline queue system using GitLab CI resource groups to limit concurrent Terraform operations to 5 per workspace. They added CloudWatch alarms for DynamoDB throttling events that trigger a PagerDuty notification when throttled requests exceed 10 in a 5-minute window.

Commands

aws dynamodb update-table --table-name terraform-state-locks --billing-mode PAY_PER_REQUEST --region us-east-1

terraform plan -lock-timeout=300s -out=capacity-fix.tfplan

aws cloudwatch put-metric-alarm --alarm-name dynamodb-terraform-locks-throttle --namespace AWS/DynamoDB --metric-name ThrottledRequests --dimensions Name=TableName,Value=terraform-state-locks --comparison-operator GreaterThanThreshold --threshold 10 --period 300 --evaluation-periods 1 --statistic Sum --alarm-actions arn:aws:sns:us-east-1:123456789012:pagerduty-critical --region us-east-1

Symptom

After upgrading the AWS provider from v5.30.0 to v5.31.0, terraform plan shows 53 unexpected in-place updates across resources that have not been modified. All changes are cosmetic attribute reorderings and default value additions, but they block the CI pipeline which enforces zero-diff plans.

Error Message

# aws_security_group.app_sg will be updated in-place
  ~ resource "aws_security_group" "app_sg" {
        id                     = "sg-0a1b2c3d4e5f67890"
        name                   = "prod-app-sg"
      ~ tags                   = {
          + "ManagedBy"        = "terraform"
            # (3 unchanged elements hidden)
        }
      ~ tags_all               = {
          + "ManagedBy"        = "terraform"
            # (3 unchanged elements hidden)
        }
      ~ ingress                = [
          ~ {
              ~ self             = false -> false
              ~ prefix_list_ids  = [] -> []
              ~ description      = "" -> "" 
                # (4 unchanged attributes hidden)
            },
        ]
        # (5 unchanged attributes hidden)
    }

Plan: 0 to add, 53 to change, 0 to destroy.

Error: Terraform plan has 53 unexpected changes. CI policy requires zero-diff
plans for production deployments. Pipeline blocked.

Root Cause

The AWS provider v5.31.0 introduced changes to how default values are handled for several resource types. Specifically, the provider now explicitly includes default values in the plan output for attributes that were previously omitted when they matched the API default. For aws_security_group resources, the provider began including default values for self (false), prefix_list_ids (empty list), and description (empty string) in ingress and egress rule blocks, even when these values had not changed. Additionally, the provider introduced a new default_tags behavior where tags propagated from the provider block's default_tags configuration are now shown as explicit changes in the plan when the resource state was written by an older provider version. This affected all 53 resources that had security group rules or were subject to default tag propagation. The CI pipeline's zero-diff policy correctly flagged these as changes because the plan file contained 53 update operations, even though applying them would result in no actual infrastructure modifications. The issue was a known provider regression documented in hashicorp/terraform-provider-aws#35421, but the team had not reviewed release notes before upgrading.

Diagnosis Steps

1. 1.terraform plan -no-color 2>&1 | grep -c 'will be updated in-place'
2. 2.terraform plan -no-color 2>&1 | grep -B5 'will be updated' | grep 'resource' | awk '{print $2}' | sort | uniq -c | sort -rn
3. 3.terraform plan -json 2>&1 | jq '[.resource_changes[] | select(.change.actions[] == "update")] | group_by(.type) | map({type: .[0].type, count: length})'
4. 4.diff <(terraform state pull | jq -S '.resources[] | select(.type=="aws_security_group") | .instances[0].attributes.ingress') <(terraform show -json | jq -S '.values.root_module.resources[] | select(.type=="aws_security_group") | .values.ingress') | head -30
5. 5.curl -s https://api.github.com/repos/hashicorp/terraform-provider-aws/issues/35421 | jq '.title, .state'

Solution

The team implemented a two-phase fix. First, to unblock the CI pipeline immediately, they applied the cosmetic changes in a dedicated commit by running terraform apply with the plan that contained the 53 updates. Since all changes were default-value additions and attribute reorderings with no actual infrastructure impact, the apply completed in under 2 minutes with no service disruption. This updated the state file to match the new provider's expectations for default values. After the apply, a subsequent terraform plan showed zero changes, confirming the diff was purely cosmetic. Second, to prevent this from recurring, the team updated their CI pipeline to run a pre-check that distinguishes between cosmetic provider diffs and actual infrastructure changes. The script parses terraform plan -json output and classifies changes into categories: actual attribute modifications, default value additions, and tag propagation updates. Only actual attribute modifications trigger the zero-diff policy violation. They also pinned the provider version more strictly and added a staging environment validation step that must pass before production provider upgrades are permitted.

Commands

terraform plan -out=cosmetic-fix.tfplan -no-color 2>&1 | tee /var/log/terraform/cosmetic-plan.log

terraform apply cosmetic-fix.tfplan

terraform plan -detailed-exitcode; echo "Exit code: $?"

terraform providers lock -platform=linux_amd64 -platform=darwin_amd64 -platform=darwin_arm64

Symptom

Terraform apply failed midway through creating a new ECS service deployment. Some resources were created in AWS but not recorded in state. Subsequent plans attempt to create duplicate resources, and manual AWS console inspection shows orphaned load balancer target groups and ECS task definitions.

Error Message

aws_ecs_task_definition.api_v2: Creating...
aws_ecs_task_definition.api_v2: Creation complete after 2s [id=arn:aws:ecs:us-east-1:123456789012:task-definition/prod-api-v2:1]
aws_lb_target_group.api_v2: Creating...
aws_lb_target_group.api_v2: Creation complete after 3s [id=arn:aws:elasticloadbalancing:us-east-1:123456789012:targetgroup/prod-api-v2-tg/abc123def456]
aws_lb_listener_rule.api_v2: Creating...
aws_ecs_service.api_v2: Creating...

Error: creating ECS Service (prod-api-v2): InvalidParameterException: Unable to
assume the service linked role. Please verify that the ECS service linked role
exists. (Service: AmazonECS; Status Code: 400; Error Code: InvalidParameterException;
Request ID: a1b2c3d4-e5f6-7890-abcd-ef1234567890)

  on modules/ecs-service/main.tf line 87, in resource "aws_ecs_service" "api_v2":
  87: resource "aws_ecs_service" "api_v2" {

aws_lb_listener_rule.api_v2: Creation complete after 1s [id=arn:aws:elasticloadbalancing:us-east-1:123456789012:listener-rule/app/prod-alb/50dc6c495c0c9188/abc123/def456]

Apply complete! Resources: 3 added, 0 changed, 0 destroyed.

The apply was interrupted and some resources may not have been recorded in state.
Please run "terraform plan" to check for any discrepancies.

Root Cause

A terraform apply was executed to deploy a new v2 API service on ECS. The apply created the ECS task definition, ALB target group, and ALB listener rule successfully, but failed when creating the ECS service because the required IAM service-linked role for ECS (AWSServiceRoleForECS) did not exist in the account. The service-linked role should have been created by a prerequisite Terraform module, but that module had been recently refactored and the role creation was accidentally removed during code cleanup. Terraform recorded the successfully created resources (task definition, target group, listener rule) in the state file, but the ECS service was not created and not recorded. However, the listener rule was created with a forward action pointing to the new target group, which had no registered targets. This caused approximately 15% of production API traffic to be routed to the empty target group, resulting in HTTP 502 errors for end users. The infrastructure was left in an inconsistent state where some resources existed in both AWS and Terraform state, the ECS service existed in neither, and the listener rule was actively routing traffic to nowhere. Subsequent terraform plan runs showed the ECS service as needing creation but did not indicate any issue with the listener rule because it was already successfully tracked in state.

Diagnosis Steps

1. 1.terraform state list | grep api_v2
2. 2.aws ecs describe-services --cluster prod-ecs-cluster --services prod-api-v2 --region us-east-1 2>&1
3. 3.aws elbv2 describe-target-health --target-group-arn arn:aws:elasticloadbalancing:us-east-1:123456789012:targetgroup/prod-api-v2-tg/abc123def456 --region us-east-1
4. 4.aws elbv2 describe-rules --listener-arn arn:aws:elasticloadbalancing:us-east-1:123456789012:listener/app/prod-alb/50dc6c495c0c9188/abc123 --region us-east-1 | jq '.Rules[] | select(.Actions[].TargetGroupArn | contains("api-v2"))'
5. 5.aws iam get-role --role-name AWSServiceRoleForECS 2>&1
6. 6.terraform plan -target=module.ecs_service_api_v2 2>&1 | head -60

Solution

The immediate priority was stopping the 502 errors by removing the listener rule that was routing traffic to the empty target group. The team ran terraform destroy -target=aws_lb_listener_rule.api_v2 to remove the problematic listener rule, which immediately restored 100% healthy traffic routing. Next, they addressed the root cause by creating the missing ECS service-linked role. They added the aws_iam_service_linked_role resource back to the prerequisite module and applied it: terraform apply -target=aws_iam_service_linked_role.ecs. With the role in place, they cleaned up the orphaned resources by first running terraform state rm on the target group and task definition, then manually deleting them from AWS using the CLI to ensure a clean slate. Finally, they ran the full terraform apply for the v2 service module, which successfully created all resources including the ECS service, and registered tasks in the target group before re-adding the listener rule. The team also added a depends_on relationship between the ECS service module and the IAM role module to prevent this ordering issue in the future.

Commands

terraform destroy -target=aws_lb_listener_rule.api_v2 -auto-approve

aws iam create-service-linked-role --aws-service-name ecs.amazonaws.com --region us-east-1

terraform state rm aws_lb_target_group.api_v2 && terraform state rm aws_ecs_task_definition.api_v2

aws elbv2 delete-target-group --target-group-arn arn:aws:elasticloadbalancing:us-east-1:123456789012:targetgroup/prod-api-v2-tg/abc123def456 --region us-east-1

terraform apply -target=module.ecs_service_api_v2 -auto-approve

Symptom

Production Terraform state file was written to an S3 bucket in the development AWS account (account 987654321098) instead of the production account (account 123456789012). Production infrastructure is unmanaged and development team discovers unknown state files in their bucket.

Error Message

Initializing the backend...

Successfully configured the backend "s3"! Terraform will automatically
use this backend unless the backend configuration changes.

Warning: Backend configuration changed

The backend configuration has changed since the last time Terraform was
initialized. Terraform has detected that the following backend attributes
have changed:
  - bucket: "prod-terraform-state-us-east-1" -> "dev-terraform-state-us-east-1"
  - role_arn: (changed)

Do you want to copy existing state to the new backend?
  Enter a value: yes

Error: Failed to read state from S3

The state file in s3://prod-terraform-state-us-east-1/infrastructure/production/terraform.tfstate
is now empty (0 bytes). The previous state has been migrated to
s3://dev-terraform-state-us-east-1/infrastructure/production/terraform.tfstate
in account 987654321098.

state serial: 4312, lineage: "f3a4b5c6-d7e8-9012-3456-789abcdef012"

Root Cause

A junior engineer was setting up a new Terraform workspace for a development environment and modified the backend.tf file to point to the development S3 bucket. They committed this change to a feature branch, but due to a misconfigured branch protection rule, the change was merged directly to the main branch without code review. The CI pipeline for the main branch ran terraform init, which detected the backend configuration change and prompted for state migration. Because the CI pipeline was configured with auto-approve flags and the -input=false flag was not set, the init process defaulted to migrating the state. The production state file containing the complete infrastructure definition for 347 resources was copied from the production S3 bucket (prod-terraform-state-us-east-1 in account 123456789012) to the development S3 bucket (dev-terraform-state-us-east-1 in account 987654321098), and the original state file in the production bucket was emptied to 0 bytes. This left all production infrastructure completely unmanaged by Terraform, and the state file containing sensitive information such as database passwords and API keys was now accessible to the entire development team in the less-restricted development account. The development team discovered the foreign state files during a routine bucket audit and escalated to the platform team.

Diagnosis Steps

1. 1.aws s3 cp s3://prod-terraform-state-us-east-1/infrastructure/production/terraform.tfstate /tmp/prod_state.json --region us-east-1 && wc -c /tmp/prod_state.json
2. 2.aws s3api list-object-versions --bucket prod-terraform-state-us-east-1 --prefix infrastructure/production/terraform.tfstate --region us-east-1 --query 'Versions[?Size>`0`] | sort_by(@, &LastModified) | [-1]'
3. 3.aws s3 cp s3://dev-terraform-state-us-east-1/infrastructure/production/terraform.tfstate /tmp/dev_state.json --region us-east-1 --profile dev-account && cat /tmp/dev_state.json | jq '.serial, .lineage, (.resources | length)'
4. 4.git log --oneline --all -- backend.tf | head -10
5. 5.aws cloudtrail lookup-events --lookup-attributes AttributeKey=EventName,AttributeValue=PutObject --start-time 2026-06-17T00:00:00Z --end-time 2026-06-18T12:00:00Z --region us-east-1 | jq '.Events[] | select(.Resources[]?.ResourceName | contains("terraform.tfstate"))'

Solution

The recovery process required careful coordination to restore the state to the correct account without losing any data. First, the team immediately disabled all CI pipelines to prevent any further Terraform operations. They retrieved the migrated state file from the development account bucket using aws s3 cp with the development account profile. They verified the state integrity by checking the serial number, lineage, and resource count (347 resources, serial 4312). Next, they restored the original backend.tf configuration by reverting the merge commit with git revert. They used aws s3 cp to upload the state file back to the production bucket in the correct account. To verify correctness, they ran terraform init followed by terraform plan, which showed zero changes, confirming the state matched the actual infrastructure. They then deleted the state file from the development account bucket and rotated all secrets that were exposed in the state file, including 12 RDS passwords, 8 API keys, and 3 TLS certificates. Branch protection rules were strengthened to require at least two approvals for changes to backend.tf, and a CODEOWNERS file was added requiring platform team review for any backend configuration changes. The CI pipeline was updated to use -input=false on all terraform init commands to prevent unattended state migrations.

Commands

aws s3 cp s3://dev-terraform-state-us-east-1/infrastructure/production/terraform.tfstate /tmp/recovered_state.json --region us-east-1 --profile dev-account

aws s3 cp /tmp/recovered_state.json s3://prod-terraform-state-us-east-1/infrastructure/production/terraform.tfstate --region us-east-1 --profile prod-account

git revert --no-edit abc123def456 && git push origin main

terraform init -input=false -reconfigure && terraform plan -detailed-exitcode

aws s3 rm s3://dev-terraform-state-us-east-1/infrastructure/production/terraform.tfstate --region us-east-1 --profile dev-account

Symptom

A production plan proposed replacing a shared security group immediately after an import cleanup.

Error Message

Error: resource already managed by Terraform

Root Cause

The operator imported a live security group to a temporary root-module address instead of the intended nested module address. Terraform state identity then disagreed with configuration. The next plan saw the real module resource as new and the temporary address as orphaned, creating dangerous replacement and destroy actions.

Diagnosis Steps

1. 1.terraform state list | grep security_group # Find current state addresses
2. 2.terraform plan -out=import-fix.tfplan # Confirm proposed destroy or replacement
3. 3.terraform state show aws_security_group.payments # Inspect imported object identity

Solution

Move the state object to the correct module address with `terraform state mv`, review the plan, and only apply after the diff is empty or expected. Do not delete and re-import blindly; that increases drift risk.

Commands

terraform state mv aws_security_group.payments module.network.aws_security_group.payments

terraform plan -out=import-fix.tfplan

terraform apply import-fix.tfplan

Symptom

Terraform plan shows phantom resources that do not exist in AWS, existing resources marked for recreation, and serial mismatch errors. Two deployments triggered within 8 seconds of each other for the payments-vpc workspace both reported success, but subsequent plans are inconsistent and show 23 resources needing recreation.

Error Message

Error: Saved plan is stale

The current state has changed since the plan was created. The plan is
no longer valid. Please create a new plan.

State serial: 1847 (expected: 1845)
State lineage: "f3a2b1c0-d4e5-6789-abcd-1234567890ef"

Error refreshing state: state snapshot contains resources that reference
providers no longer available in configuration:
  - registry.terraform.io/hashicorp/aws (3 resources orphaned)

Additionally: duplicate resource entry detected:
  aws_security_group.payments_ingress appears 2 times in state

Root Cause

Two GitHub Actions workflows triggered simultaneously when a merge queue processed two pull requests within an 8-second window against the payments-vpc workspace. Both pipelines ran terraform plan concurrently, each reading state serial 1845 from S3. Pipeline A acquired the DynamoDB lock first, applied its changes (adding 3 security group rules), and wrote state serial 1846 before releasing the lock. Pipeline B then acquired the lock and attempted to apply its cached plan which was generated against serial 1845. The plan included modifications to the same security groups that Pipeline A had already changed. Because Pipeline B's plan file was generated before Pipeline A's changes, it wrote a state that partially overwrote Pipeline A's resource entries while adding its own. The resulting state serial 1847 contained duplicate entries for aws_security_group.payments_ingress with conflicting attribute values, three orphaned provider references from a partial write failure, and missing dependency metadata for 6 subnet associations. The fundamental issue was that DynamoDB locking only protects the write phase, not the plan-read-apply cycle as an atomic unit. The plan was generated outside the lock window, creating a classic time-of-check-to-time-of-use vulnerability.

Diagnosis Steps

1. 1.aws s3api list-object-versions --bucket acme-terraform-state-prod --prefix payments-vpc/terraform.tfstate --max-items 5 --region us-east-1 | jq '.Versions[] | {VersionId, LastModified, Size}'
2. 2.terraform state pull | jq '.resources | group_by(.type + "." + .name) | map(select(length > 1)) | .[][] | {type, name, instances: (.instances | length)}'
3. 3.aws dynamodb scan --table-name acme-terraform-locks --filter-expression 'contains(LockID, :ws)' --expression-attribute-values '{":ws": {"S": "payments-vpc"}}' --region us-east-1
4. 4.gh run list --workflow=terraform-payments-vpc.yml --limit=10 --json databaseId,status,createdAt,conclusion | jq '.[] | select(.createdAt > "2026-03-14T10:00:00")'
5. 5.aws s3api get-object --bucket acme-terraform-state-prod --key payments-vpc/terraform.tfstate --version-id VERSION_BEFORE_CORRUPTION /tmp/clean_state.json --region us-east-1 && jq '.serial, .lineage, (.resources | length)' /tmp/clean_state.json

Solution

Recovery required rolling back to the last known good state version from S3 versioning. The team first disabled all CI pipelines for the payments-vpc workspace by adding a branch protection hold. They identified the clean state version (serial 1845) using aws s3api list-object-versions and downloaded it. After verifying the clean state had the correct resource count (47 resources vs the corrupted 53), they ran terraform state push -force /tmp/clean_state.json to overwrite the corrupted state. A terraform plan -refresh-only then reconciled the state with actual AWS resources, picking up the changes from both Pipeline A and Pipeline B that had actually been applied to AWS. They ran terraform apply -refresh-only to update the state to match reality without making infrastructure changes. To prevent recurrence, they implemented a GitHub Actions concurrency group with cancel-in-progress: false that ensures only one workflow runs per workspace, and added a pre-plan script that verifies the state serial matches the expected value before generating a plan.

Commands

aws s3api get-object --bucket acme-terraform-state-prod --key payments-vpc/terraform.tfstate --version-id QUpbz3m4L9OHK7yZSnGh2kV /tmp/clean_state.json --region us-east-1

terraform state push -force /tmp/clean_state.json

terraform plan -refresh-only -out=reconcile.tfplan 2>&1 | tee /var/log/terraform/reconcile.log

terraform apply -refresh-only reconcile.tfplan

Symptom

After a routine CI pipeline run on Monday morning, all new ECS tasks in the payments-api cluster failed to start with connection timeouts to the RDS database. Existing tasks continued working. No Terraform code had changed — the only difference was a provider version bump merged the previous Friday.

Error Message

aws_security_group.payments_ecs_tasks: Modifying... [id=sg-0f7a2b3c4d5e6f789]
aws_security_group.payments_ecs_tasks: Modifications complete after 4s

~ resource "aws_security_group" "payments_ecs_tasks" {
      id                     = "sg-0f7a2b3c4d5e6f789"
      name                   = "payments-ecs-tasks-prod"
    ~ revoke_rules_on_delete = false -> true  # CHANGED BY PROVIDER DEFAULT
    ~ egress {               # DEFAULT EGRESS RULE REMOVED
        - cidr_blocks = ["0.0.0.0/0"]
        - from_port   = 0
        - protocol    = "-1"
        - to_port     = 0
      }
  }

Plan: 0 to add, 14 to change, 0 to destroy.

Root Cause

The team upgraded the AWS provider from version 5.29.0 to 5.30.0 as part of a routine monthly upgrade cycle. The changelog mentioned 'improved security group resource handling' but did not clearly call out that the default behavior for egress rules had changed. In provider 5.29.0, security groups created without explicit egress blocks retained the AWS default egress rule allowing all outbound traffic. In 5.30.0, the provider adopted a more opinionated stance: security groups without explicit egress blocks had their default egress rule removed during the next apply because the provider now treats the absence of an egress block as 'no egress rules desired' rather than 'leave AWS defaults in place.' The terraform plan showed 14 security groups being modified, but the change was a single boolean flip (revoke_rules_on_delete) and an egress rule removal that appeared minor. The reviewing engineer approved the plan without realizing the egress removal would cut off all outbound connectivity for ECS tasks, preventing them from reaching the RDS endpoint at port 5432. Existing tasks were unaffected because security group changes only apply to new network connections, not established ones. New ECS task launches failed immediately because they could not establish outbound connections to RDS, ElastiCache, or external payment processor APIs.

Diagnosis Steps

1. 1.aws ec2 describe-security-groups --group-ids sg-0f7a2b3c4d5e6f789 --region us-east-1 --query 'SecurityGroups[0].{Ingress: IpPermissions, Egress: IpPermissionsEgress}'
2. 2.aws ecs describe-tasks --cluster payments-api-prod --tasks $(aws ecs list-tasks --cluster payments-api-prod --desired-status STOPPED --max-items 5 --query 'taskArns' --output text) --query 'tasks[].stoppedReason'
3. 3.git log --oneline -10 -- '*.tf' '*.lock.hcl' | head -10
4. 4.diff <(git show HEAD~1:.terraform.lock.hcl) .terraform.lock.hcl | grep -A2 -B2 'version'

Solution

The immediate fix was to explicitly add egress rules to all 14 affected security groups. The team added egress blocks to every aws_security_group resource that previously relied on the AWS default, explicitly defining the all-traffic outbound rule. They ran terraform plan to verify the egress rules would be re-added and applied the change, restoring outbound connectivity within 12 minutes of diagnosis. New ECS tasks launched successfully after the security groups were updated. For the longer term, the team pinned the AWS provider to exact versions using the = operator instead of ~> and established a provider upgrade runbook. The runbook requires: reading the full changelog including the Upgrade Guide section, running plan against a staging environment first and waiting 24 hours before production, and specifically reviewing any security group changes line by line. They also added a CI check that diffs the terraform plan output for any security group egress modifications and flags them for mandatory senior engineer review.

Commands

terraform plan -target=aws_security_group.payments_ecs_tasks -out=fix-sg.tfplan

terraform apply fix-sg.tfplan

aws ec2 describe-security-groups --group-ids sg-0f7a2b3c4d5e6f789 --query 'SecurityGroups[0].IpPermissionsEgress' --region us-east-1

aws ecs update-service --cluster payments-api-prod --service payments-api --force-new-deployment --region us-east-1

Symptom

A production EC2 instance running the order-processing service was terminated and replaced during a terraform apply. The replacement instance came up with a fresh root volume, losing 6 hours of local SQS dead-letter queue processing state, active network connections, and the instance's IAM instance profile association took 4 minutes to propagate, causing a processing gap.

Error Message

  # aws_instance.main will be destroyed
  - resource "aws_instance" "main" {
      - ami                    = "ami-0abc123def456789a" -> null
      - id                     = "i-0f7a2b3c4d5e6f789" -> null
      - instance_type          = "c6i.2xlarge" -> null
      - private_ip             = "10.100.47.23" -> null
      - tags                   = {
          - "Name" = "order-processor-prod-1a"
        } -> null
    }

  # aws_instance.order_processor will be created
  + resource "aws_instance" "order_processor" {
      + ami                    = "ami-0abc123def456789a"
      + instance_type          = "c6i.2xlarge"
      + id                     = (known after apply)
      + private_ip             = (known after apply)
    }

Plan: 1 to add, 0 to change, 1 to destroy.

Root Cause

An engineer refactored the Terraform configuration to use more descriptive resource names as part of a code quality initiative. The resource aws_instance.main was renamed to aws_instance.order_processor to better reflect its purpose. The engineer was unaware of Terraform's moved block feature introduced in version 1.1 and assumed that renaming a resource in the HCL code would simply update the state reference, similar to how renaming a variable works in most programming languages. Without a moved block, Terraform interpreted the change as: the resource aws_instance.main no longer exists in configuration (destroy it) and a new resource aws_instance.order_processor needs to be created. The plan clearly showed '1 to add, 1 to destroy' but the reviewing engineer approved it believing it was adding a new instance alongside the existing one, not realizing 'main' and 'order_processor' referred to the same production box. The destroyed instance held local state for an SQS dead-letter queue processor that stored retry metadata on the EBS root volume. The new instance launched with a fresh root volume, losing the retry state and causing 847 payment processing messages to be permanently lost from the local retry queue.

Diagnosis Steps

1. 1.terraform plan 2>&1 | grep -E '(will be destroyed|will be created|must be replaced)' | head -20
2. 2.terraform state list | grep -i 'instance\|ec2'
3. 3.git diff HEAD~1 -- '*.tf' | grep -E '^[+-].*resource.*aws_instance'
4. 4.aws ec2 describe-instances --instance-ids i-0f7a2b3c4d5e6f789 --region us-east-1 --query 'Reservations[0].Instances[0].{State: State.Name, LaunchTime: LaunchTime, StateTransitionReason: StateTransitionReason}'

Solution

The destroyed instance could not be recovered, so the team focused on restoring service. The newly created aws_instance.order_processor was already running with the correct AMI and instance type. The team reattached the EBS data volume from a snapshot taken 6 hours prior, restored the SQS retry metadata from the snapshot, and reprocessed failed messages from the SQS dead-letter queue. To fix the root cause and prevent recurrence, the team added a mandatory CI check that runs a custom script parsing terraform plan -json output for any destroy operations on aws_instance, aws_db_instance, aws_eks_cluster, and aws_s3_bucket resource types. If any of these critical resource types appear in the destroy list, the pipeline fails with an error message requiring either a moved block or explicit approval from two senior engineers. They also added lifecycle prevent_destroy to all stateful EC2 instances.

Commands

terraform state show aws_instance.order_processor

aws ec2 describe-volumes --filters Name=attachment.instance-id,Values=i-0f7a2b3c4d5e6f789 --query 'Volumes[*].{VolumeId:VolumeId,State:State}' --region us-east-1

terraform plan -json | jq '.resource_changes[] | select(.change.actions | contains(["delete"])) | {address, type, change: .change.actions}'

Symptom

A terraform apply deleted 3 custom security group rules, removed a manually-added NAT gateway route, and reverted an RDS parameter group change that had been keeping the payments database stable for 4 months. The payments API immediately began experiencing connection pooling errors and 2x latency increase.

Error Message

  # aws_security_group_rule.payments_db_custom_cidr will be destroyed
  # (not in configuration)
  - resource "aws_security_group_rule" "payments_db_custom_cidr" {
      - cidr_blocks       = ["10.200.0.0/16"] -> null
      - from_port         = 5432 -> null
      - protocol          = "tcp" -> null
      - security_group_id = "sg-0db5a4c3e2f1a0987" -> null
      - to_port           = 5432 -> null
      - type              = "ingress" -> null
    }

  # aws_db_parameter_group.payments_prod will be updated in-place
  ~ resource "aws_db_parameter_group" "payments_prod" {
      ~ parameter {
          ~ name  = "max_connections"
          ~ value = "500" -> "200"  # REVERTING MANUAL CHANGE
        }
    }

Plan: 0 to add, 3 to change, 3 to destroy.

Root Cause

Four months earlier, the payments team experienced an urgent production incident where the RDS database ran out of connections. The on-call DBA increased max_connections from 200 to 500 directly in the AWS console and added a security group rule to allow temporary access from a debugging VPC (10.200.0.0/16). A NAT gateway route was also added manually to enable a new microservice to reach an external payment processor. All three changes were made as emergency fixes with the intention of 'codifying them in Terraform later' — but no follow-up tickets were created and the changes were never committed to the Terraform configuration. The team had no drift detection mechanism: no scheduled terraform plan runs, no AWS Config rules comparing actual vs desired state, and no regular reconciliation process. Four months later, a junior engineer made an unrelated change to add a tag to the VPC and ran terraform apply. Terraform refreshed the state, detected that the console changes did not match the configuration, and dutifully reverted all manual changes to match the declared configuration. The max_connections drop from 500 to 200 immediately caused connection pool exhaustion, and the removed security group rule cut off the debugging VPC access that had quietly become a production dependency.

Diagnosis Steps

1. 1.terraform plan -refresh-only 2>&1 | grep -E '(has changed|has been deleted|drift detected)'
2. 2.aws rds describe-db-parameters --db-parameter-group-name payments-prod-pg15 --source user --region us-east-1 --query 'Parameters[?ParameterName==`max_connections`]'
3. 3.aws cloudtrail lookup-events --lookup-attributes AttributeKey=ResourceType,AttributeValue=AWS::RDS::DBParameterGroup --start-time 2026-02-01T00:00:00Z --end-time 2026-06-18T00:00:00Z --region us-east-1 --max-items 20
4. 4.terraform state show aws_db_parameter_group.payments_prod | grep max_connections

Solution

The team immediately reverted the terraform apply by restoring the previous state version from S3 and running a targeted apply. They then codified all three manual changes into the Terraform configuration: updated the RDS parameter group to set max_connections to 500, added the security group rule for the 10.200.0.0/16 CIDR, and added the NAT gateway route for the payment processor. After applying these changes, terraform plan showed zero differences, confirming full reconciliation. To prevent recurrence, they implemented a weekly scheduled GitHub Actions workflow that runs terraform plan -detailed-exitcode against every workspace and posts drift reports to a dedicated Slack channel. Any detected drift is automatically converted to a Jira ticket assigned to the owning team. They also enabled AWS Config drift detection rules for critical resource types and configured CloudTrail alerts for any console-based modifications to Terraform-managed resources.

Commands

terraform plan -refresh-only -detailed-exitcode 2>&1 | tee drift-report.log; echo "Exit code: $?"

terraform apply -refresh-only -auto-approve

aws configservice get-compliance-details-by-resource --resource-type AWS::RDS::DBParameterGroup --resource-id payments-prod-pg15 --region us-east-1

terraform import aws_security_group_rule.payments_db_debug_vpc sg-0db5a4c3e2f1a0987_ingress_tcp_5432_5432_10.200.0.0/16

Symptom

Every Terraform operation across all CI pipelines for the prod-eks workspace returns 'Error acquiring the state lock' and no deployments can proceed. The lock has been held for 7 hours with no active Terraform process. On-call pages are firing because a critical EKS node group scaling change cannot be deployed during a traffic spike.

Error Message

Error: Error acquiring the state lock

Error message: ConditionalCheckFailedException: The conditional request failed
Lock Info:
  ID:        e7f8a9b0-c1d2-3456-7890-abcdef123456
  Path:      acme-terraform-state-prod/prod-eks/terraform.tfstate
  Operation: OperationTypeApply
  Who:       runner@github-actions-runner-7f8b9c-xk4wd
  Version:   1.7.4
  Created:   2026-04-22 01:47:12.338291 +0000 UTC
  Info:

Terraform acquires a state lock to protect against two processes
writing to state simultaneously. The lock has been held for 7h14m
and may have been left behind by a crashed process.

Root Cause

A GitHub Actions self-hosted runner on an EC2 instance was executing a terraform apply for the prod-eks workspace when the underlying EC2 instance experienced a hardware failure and was terminated by AWS. The instance received a scheduled event notification but the runner process did not have a graceful shutdown handler configured for EC2 instance state change events. When the instance was hard-terminated, the Terraform process was killed with SIGKILL, which cannot be caught or handled by any process. Terraform's normal shutdown sequence — which includes releasing the DynamoDB state lock — never executed. The DynamoDB lock record remained in the acme-terraform-locks table with lock ID e7f8a9b0-c1d2-3456-7890-abcdef123456, and since the process that created it no longer existed on any machine, no process could release it through normal means. All subsequent Terraform operations including plan, apply, state list, and even output for the prod-eks workspace were blocked because every operation requires acquiring the state lock. The monitoring system only alerted on failed applies but had no check for lock age, so the stale lock went undetected for 7 hours until the morning shift tried to deploy an urgent EKS node scaling change.

Diagnosis Steps

1. 1.aws dynamodb get-item --table-name acme-terraform-locks --key '{"LockID": {"S": "acme-terraform-state-prod/prod-eks/terraform.tfstate"}}' --region us-east-1 | jq '.Item'
2. 2.aws ec2 describe-instance-status --instance-ids i-0runner7f8b9cxk4w --include-all-instances --region us-east-1 --query 'InstanceStatuses[0].{State: InstanceState.Name, Events: Events}'
3. 3.terraform force-unlock -dry-run e7f8a9b0-c1d2-3456-7890-abcdef123456 2>&1
4. 4.aws dynamodb scan --table-name acme-terraform-locks --region us-east-1 --select COUNT

Solution

The on-call engineer first verified that no active Terraform process held the lock by confirming the runner EC2 instance was in terminated state and checking all other runners for active Terraform processes. After confirming the lock was orphaned, they executed terraform force-unlock e7f8a9b0-c1d2-3456-7890-abcdef123456 which removed the stale DynamoDB record. They then ran terraform plan to verify the state file was not corrupted by the interrupted apply. The plan showed 2 resources with partial updates that needed completion — an EKS node group that was mid-scaling and an IAM policy that was partially attached. A terraform apply completed these partial changes cleanly. To prevent recurrence, the team deployed a Lambda function triggered by a CloudWatch Events rule every 15 minutes that scans the DynamoDB lock table for any locks older than 45 minutes. The Lambda sends a PagerDuty alert with the lock details and optionally auto-releases locks older than 2 hours after confirming the holding process is dead. They also migrated CI runners to use AWS-managed GitHub Actions runners with built-in health checks.

Commands

terraform force-unlock e7f8a9b0-c1d2-3456-7890-abcdef123456

terraform plan -out=recovery.tfplan 2>&1 | tee /tmp/post-unlock-plan.log

terraform apply recovery.tfplan

aws dynamodb scan --table-name acme-terraform-locks --filter-expression 'attribute_exists(Info)' --region us-east-1 --output table

Symptom

Production PostgreSQL RDS instance (db.r6g.4xlarge, 2.3TB data) was destroyed during what was intended as a staging environment teardown. All customer-facing services returned HTTP 500 errors. Data loss confirmed for the 22 minutes between the last automated backup and the destroy event.

Error Message

aws_db_instance.primary: Destroying... [id=prod-payments-db-us-east-1]
aws_db_instance.primary: Still destroying... [5m10s elapsed]
aws_db_instance.primary: Still destroying... [10m20s elapsed]
aws_db_instance.primary: Destruction complete after 14m47s

aws_db_subnet_group.primary: Destroying... [id=prod-payments-db-subnet-group]
aws_db_subnet_group.primary: Destruction complete after 2s

Destroy complete! Resources: 31 destroyed.

Root Cause

An engineer was tasked with tearing down a staging environment that was no longer needed. The team used Terraform workspaces to manage environment isolation within a single configuration, with workspace names 'staging' and 'production'. The engineer opened two terminal tabs — one connected to staging and one to production from an earlier debugging session. The engineer typed terraform workspace select staging in what they believed was the correct terminal tab, but the command was actually entered in the production tab where the workspace was already set to production. The workspace select command returned 'Already on workspace production' but the engineer did not read the output, assuming the select succeeded. They then ran terraform destroy -auto-approve and approved the destruction of 31 production resources including the primary RDS instance, its read replica, 4 ElastiCache nodes, and the associated networking resources. The production workspace had no lifecycle prevent_destroy on the RDS instance, no OPA/Sentinel policy blocking destroy operations, and the -auto-approve flag bypassed the interactive confirmation prompt that would have displayed the workspace name and resource identifiers. The engineer only realized the mistake when PagerDuty alerts fired 30 seconds after the RDS instance entered 'deleting' state.

Diagnosis Steps

1. 1.terraform workspace show
2. 2.aws rds describe-db-instances --db-instance-identifier prod-payments-db-us-east-1 2>&1 | head -5
3. 3.aws rds describe-db-snapshots --db-instance-identifier prod-payments-db-us-east-1 --snapshot-type automated --query 'sort_by(DBSnapshots, &SnapshotCreateTime)[-1].{SnapshotId: DBSnapshotIdentifier, Created: SnapshotCreateTime}' --region us-east-1
4. 4.aws cloudtrail lookup-events --lookup-attributes AttributeKey=EventName,AttributeValue=DeleteDBInstance --start-time $(date -u -d '2 hours ago' +%Y-%m-%dT%H:%M:%SZ) --region us-east-1

Solution

The team immediately began RDS restoration from the most recent automated snapshot, which was taken 22 minutes before the destroy. They executed aws rds restore-db-instance-from-db-snapshot with the original instance class, parameter group, and subnet group configuration. While the database was restoring (which took 45 minutes for the 2.3TB instance), they used application-level write-ahead logs from the payments service to identify 1,247 transactions that occurred in the 22-minute gap. After the database was online, they replayed these transactions from the application's event store, recovering all but 3 transactions that had no event log entries. The full infrastructure was then rebuilt using terraform apply against the production workspace, and terraform import was used for the restored RDS instance. Post-incident, the team implemented five changes: lifecycle prevent_destroy on all RDS and ElastiCache resources, an OPA policy rejecting terraform destroy on production workspaces, removal of all uses of -auto-approve in CI scripts, migration from workspaces to separate AWS accounts for environment isolation, and a shell prompt customization that displays the current Terraform workspace in bold red text when set to production.

Commands

aws rds restore-db-instance-from-db-snapshot --db-instance-identifier prod-payments-db-us-east-1 --db-snapshot-identifier rds:prod-payments-db-us-east-1-2026-04-22-01-30 --db-instance-class db.r6g.4xlarge --db-subnet-group-name prod-payments-db-subnet-group --vpc-security-group-ids sg-0db5a4c3e2f1a0987 --region us-east-1

terraform workspace select production && terraform plan -out=restore.tfplan

terraform import aws_db_instance.primary prod-payments-db-us-east-1

Symptom

Running terraform plan against the platform stack takes over 30 minutes and never completes, consuming 100% CPU on the CI runner. The process must be manually killed. When run with TF_LOG=DEBUG, the log shows repeated evaluation cycles between the networking and compute modules, each iteration generating identical intermediate values without converging.

Error Message

Error: Cycle: module.networking.aws_security_group.eks_cluster, module.compute.aws_eks_cluster.prod, module.networking.aws_security_group_rule.eks_to_rds, module.compute.aws_security_group.eks_nodes, module.networking.data.aws_security_group.eks_nodes

This error indicates a dependency cycle in the configuration. Terraform cannot determine the correct order to create or update these resources because they form a circular reference chain.

Root Cause

The platform team had two modules — networking and compute — that developed a circular dependency over several months of incremental changes. The networking module created a security group for the EKS cluster (eks_cluster_sg) and needed the EKS node security group ID from the compute module to create an ingress rule allowing node-to-control-plane communication. Meanwhile, the compute module needed the EKS cluster security group ID from the networking module to configure the cluster's vpc_config. Over time, additional cross-references were added: the networking module added a data source looking up the EKS node security group by tag, and the compute module referenced a security group rule from the networking module as a dependency for the node group. This created a four-resource cycle: networking creates SG A, compute needs SG A to create EKS cluster, EKS cluster creates SG B for nodes, networking needs SG B to create SG rule, but SG rule is in the same module as SG A. Terraform's dependency resolver detected the cycle and either threw an explicit error (in version 1.7+) or entered an infinite evaluation loop (in older versions with certain provider combinations). The root issue was that the module boundaries did not align with resource dependency boundaries — resources that depend on each other should live in the same module or use explicit dependency injection rather than cross-module data source lookups.

Diagnosis Steps

1. 1.terraform plan 2>&1 | grep -i 'cycle\|circular\|loop'
2. 2.terraform graph | dot -Tsvg > /tmp/dependency-graph.svg && open /tmp/dependency-graph.svg
3. 3.terraform graph | grep -E '(module.networking.*module.compute|module.compute.*module.networking)'
4. 4.TF_LOG=TRACE terraform plan 2>&1 | grep -c 'Re-evaluating' | tail -1

Solution

The team broke the circular dependency by restructuring the module boundaries. They extracted the shared security group definitions into a new dedicated security-groups module that both networking and compute modules consume as inputs rather than creating cross-module data source lookups. The security-groups module creates all security groups and rules upfront, then passes the group IDs as outputs to the networking and compute modules via input variables. This established a clear one-directional dependency chain: security-groups module runs first, then networking and compute can run in parallel since they both only depend on security-groups outputs, not on each other. They also replaced all cross-module data source lookups with explicit variable passing through the root module, making dependencies visible in the module call blocks rather than hidden in data source filters. After the refactor, terraform plan completed in 2 minutes 14 seconds instead of hanging indefinitely, and the dependency graph showed a clean DAG with no cycles.

Commands

terraform graph 2>&1 | head -50

terraform validate 2>&1 | grep -i cycle

terraform plan -parallelism=1 2>&1 | tail -20

Symptom

A security scan flagged the production RDS master password appearing in plaintext in 47 GitHub Actions workflow logs spanning the past 3 months. The password was visible in terraform plan and terraform output command outputs that were streamed to CI logs without redaction.

Error Message

Changes to Outputs:
  + db_connection_string = "postgresql://payments_admin:Pr0d-P@yments-DB-2026!xK9mZ@prod-payments-db.c9abc123.us-east-1.rds.amazonaws.com:5432/payments"
  + rds_master_password  = "Pr0d-P@yments-DB-2026!xK9mZ"

Apply complete! Resources: 0 added, 0 changed, 0 destroyed.

Outputs:

db_connection_string = "postgresql://payments_admin:Pr0d-P@yments-DB-2026!xK9mZ@prod-payments-db.c9abc123.us-east-1.rds.amazonaws.com:5432/payments"
rds_master_password = "Pr0d-P@yments-DB-2026!xK9mZ"

Root Cause

The database module defined two outputs — rds_master_password and db_connection_string — that contained the production RDS master password. Neither output was marked with the sensitive = true flag. In Terraform, outputs without the sensitive flag are printed in plaintext during plan, apply, and when running terraform output. The CI pipeline ran terraform apply followed by terraform output to capture connection details for downstream deployment steps, and both commands printed the password in plaintext. GitHub Actions logs are retained for 90 days by default and are accessible to anyone with repository read access, which included 34 engineers, 3 contractors, and 2 automated service accounts. The password had been exposed in 47 separate workflow runs over 3 months before a security audit using trufflehog detected the credential in the logs. Additionally, the database connection string output concatenated the password inline, meaning even if rds_master_password had been marked sensitive, the password would still leak through db_connection_string because Terraform's sensitivity tracking does not automatically propagate through string interpolation in all cases. The root issue was twofold: missing sensitive flags on outputs and a design that passed passwords through Terraform outputs at all instead of using a secrets manager.

Diagnosis Steps

1. 1.terraform output -json | jq 'to_entries[] | select(.value.sensitive == false) | .key'
2. 2.grep -rn 'output.*password\|output.*secret\|output.*key\|output.*token' --include='*.tf' .
3. 3.gh run view --log 12345678 | grep -i 'password\|secret\|credential'
4. 4.trufflehog git https://github.com/acme/infrastructure --only-verified --json | jq '.SourceMetadata'

Solution

The team executed an immediate incident response. First, they rotated the RDS master password using aws rds modify-db-instance --master-user-password, generating a new 32-character random password stored directly in AWS Secrets Manager. They deleted all 47 affected GitHub Actions log runs using the gh API to remove the logs from GitHub's storage. They then refactored the Terraform configuration to stop passing the database password through outputs entirely. Instead, the RDS module generates a random password using the random_password resource, stores it directly in AWS Secrets Manager using aws_secretsmanager_secret_version, and outputs only the Secrets Manager ARN (not sensitive). Application deployments retrieve the password at runtime from Secrets Manager using IAM role-based access. For any outputs that must remain sensitive (like private keys during bootstrapping), they added sensitive = true and configured the CI pipeline to redirect terraform output to /dev/null, using terraform output -raw -json piped through a sanitization script for specific values needed downstream.

Commands

aws rds modify-db-instance --db-instance-identifier prod-payments-db-us-east-1 --master-user-password $(openssl rand -base64 32) --apply-immediately --region us-east-1

aws secretsmanager update-secret --secret-id prod/payments-db/master-password --secret-string $(aws rds describe-db-instances --db-instance-identifier prod-payments-db-us-east-1 --query 'DBInstances[0].MasterUserPassword' --output text) --region us-east-1

for run_id in $(gh run list --workflow=terraform-deploy.yml --limit=50 --json databaseId --jq '.[].databaseId'); do gh api repos/acme/infrastructure/actions/runs/$run_id/logs -X DELETE; done

Symptom

9:42 AM — the platform team's #infra-alerts channel lit up with four separate Jenkins pipeline failures within 20 minutes, each for a different microservice repo, all failing at the exact same step: the terraform apply stage. Every failure showed the identical error: 'Error acquiring the state lock.' No infrastructure changes were actually in progress at the time.

Error Message

Error: Error acquiring the state lock

Error message: ConditionalCheckFailedException: The conditional request failed
Lock Info:
  ID:        8e3f2a91-4b7d-4c6e-9a2f-1d5c7e8b0a3f
  Path:      payments-infra/terraform.tfstate
  Operation: OperationTypeApply
  Who:       jenkins@ci-agent-7
  Version:   1.7.5
  Created:   2026-07-04 06:17:03.881442 UTC

Root Cause

A pull request's Jenkins build had been manually cancelled at 6:17 AM by an engineer who realized they'd targeted the wrong branch, right as the terraform apply step was mid-flight writing changes to the ECS task definitions. Jenkins sent a SIGTERM to the build agent, which propagated to the terraform process; when that didn't stop it within the configured grace period, Jenkins escalated to SIGKILL. Terraform's lock-release logic runs as part of its normal shutdown path — it's a deferred cleanup step, not a signal handler — so a hard kill skips it entirely, leaving the DynamoDB lock item behind with no owner still running to clear it. Because all 50 microservice repos in this org share a single remote backend configuration pointing at the same DynamoDB lock table (a cost-saving decision made when the platform had 5 services, never revisited at 50), the orphaned lock on the payments-infra state file was itself scoped correctly to that one state path — but four other unrelated pipelines failed at the same moment because their scheduled nightly drift-detection plans also hash to lock-check logic that, due to a bug in the shared CI wrapper script, retried against the wrong lock path under high concurrency, surfacing the same error message across seemingly unrelated services.

Diagnosis Steps

1. 1.Run `terraform plan` to reproduce the failure locally and capture the full Lock Info block, especially the Lock ID and 'Who' field.
2. 2.Check Jenkins build history for the affected job around the timestamp in the lock's 'Created' field: `curl -s -u $USER:$TOKEN "$JENKINS_URL/job/payments-infra-apply/api/json?tree=builds[number,building,result,timestamp]"`
3. 3.Query the DynamoDB lock table directly to confirm only one stale item exists and no other legitimate lock is present: `aws dynamodb scan --table-name terraform-locks --filter-expression "LockID = :p" --expression-attribute-values '{":p":{"S":"payments-infra/terraform.tfstate"}}'`
4. 4.Confirm no Jenkins agent currently shows a running build for any pipeline touching this state path before taking any unlock action.

Solution

First, message the team to confirm nobody believes an apply is genuinely in progress — this is the single most important step, because force-unlocking a state that's actually being written to is how you get real corruption, not just an inconvenient error. Once confirmed safe, run terraform force-unlock with the exact Lock ID captured from the error output; never guess or reuse an old ID from a previous incident. After the lock clears, do not immediately resume the queued pipeline runs — run terraform plan by hand first, because the killed apply may have completed some AWS API calls (like updating the ECS task definition) before the process died, and the state file might be behind reality. In this incident, the plan showed the ECS task definition revision had already updated in AWS but the state file still pointed at the prior revision, requiring a targeted terraform apply -refresh-only to pull the true value back into state before resuming normal operations. Do not skip this reconciliation step and simply re-run the original apply — doing so risks reverting the task definition to an older revision if the state and reality disagree on direction.

Commands

terraform force-unlock 8e3f2a91-4b7d-4c6e-9a2f-1d5c7e8b0a3f

terraform apply -refresh-only

terraform plan -out=tfplan.verify

terraform show tfplan.verify | grep -A3 aws_ecs_task_definition