A CloudFormation stack update failed and rolled back, but the rollback itself also failed, leaving the stack in UPDATE_ROLLBACK_FAILED state. How do you diagnose and recover from this situation?
Quick Answer
Identify the resource causing the rollback failure from stack events, fix the underlying issue (often a resource that was manually deleted or modified), then use `continue-update-rollback` with `--resources-to-skip` to skip the problematic resource and complete the rollback. As a last resort, delete the stack with `--retain-resources`.
Detailed Answer
Understanding UPDATE_ROLLBACK_FAILED
This state occurs when CloudFormation tries to roll back a failed update but encounters an error during the rollback itself. Common causes: a resource was manually deleted outside CloudFormation and can't be restored, an IAM permission was removed, a resource dependency changed, or the rollback requires creating a resource that conflicts with an existing one. The stack is now stuck and no further updates are possible until the rollback completes.
Diagnosis Steps
Examine stack events to find the exact resource and error message causing the rollback failure. Look for events with UPDATE_ROLLBACK_FAILED or DELETE_FAILED status reason. Common error messages include 'Resource X does not exist', 'Access Denied', or 'Limit exceeded'. Cross-reference with CloudTrail to see if someone modified resources outside CloudFormation.
Recovery with continue-update-rollback
Once you understand which resources are blocking the rollback, use continue-update-rollback with --resources-to-skip to tell CloudFormation to ignore those resources and complete the rollback. Skipped resources will be removed from CloudFormation's state management (they become unmanaged). After the rollback completes, you can import them back into the stack or manage them separately.
Prevention
Protect critical resources with DeletionPolicy: Retain and UpdateReplacePolicy: Retain. Use stack policies to prevent accidental updates to sensitive resources. Implement SCP policies that prevent manual modification of CloudFormation-managed resources. Always test changes in a non-production environment first.
Code Example
# Step 1: Diagnose - find the failing resource
aws cloudformation describe-stack-events \
--stack-name production-app \
--query 'StackEvents[?ResourceStatus==`UPDATE_ROLLBACK_FAILED` || ResourceStatus==`DELETE_FAILED`].{Time:Timestamp,Resource:LogicalResourceId,Type:ResourceType,Status:ResourceStatus,Reason:ResourceStatusReason}' \
--output table
# Example output:
# Resource: ApiGatewayDeployment
# Reason: "The REST API doesn't exist (Service: AmazonApiGateway; Status Code: 404)"
# -> Someone deleted the API Gateway outside CloudFormation
# Step 2: Attempt to fix the root cause
# If the resource was manually deleted, sometimes you can recreate it
# If it's an IAM issue, fix the permissions
# Step 3: Continue rollback, skipping the problematic resource
aws cloudformation continue-update-rollback \
--stack-name production-app \
--resources-to-skip ApiGatewayDeployment ApiGatewayStage
# Step 4: Verify stack is back to UPDATE_ROLLBACK_COMPLETE
aws cloudformation describe-stacks \
--stack-name production-app \
--query 'Stacks[0].{Status:StackStatus,Reason:StackStatusReason}'
# Step 5: Now you can update the stack again
# Fix the template to handle the missing resources
aws cloudformation update-stack \
--stack-name production-app \
--template-url https://s3.amazonaws.com/templates/app-fixed.yaml \
--parameters ParameterKey=Environment,UsePreviousValue=true
# Prevention: Stack policy to protect critical resources
aws cloudformation set-stack-policy \
--stack-name production-app \
--stack-policy-body '{
"Statement": [
{
"Effect": "Deny",
"Action": "Update:Replace",
"Principal": "*",
"Resource": "LogicalResourceId/ProductionDatabase"
},
{
"Effect": "Deny",
"Action": "Update:Delete",
"Principal": "*",
"Resource": "LogicalResourceId/ProductionDatabase"
},
{
"Effect": "Allow",
"Action": "Update:*",
"Principal": "*",
"Resource": "*"
}
]
}'
# Prevention: DeletionPolicy in template
Resources:
ProductionDatabase:
Type: AWS::RDS::DBInstance
DeletionPolicy: Retain
UpdateReplacePolicy: Retain
Properties:
DBInstanceClass: db.r5.2xlarge
Engine: postgres
MultiAZ: true
DeletionProtection: true
# Nuclear option: Delete stack but retain critical resources
aws cloudformation delete-stack \
--stack-name production-app \
--retain-resources ProductionDatabase ProductionVPCInterview Tip
This is a very common production scenario. Walk through the diagnostic workflow step by step: events, identify the failing resource, understand why, fix or skip. Emphasize `--resources-to-skip` as the primary recovery tool. Mention that skipped resources become unmanaged - this has implications.