Loading...
CloudFormation
Everything for CloudFormation in one place — pick a section below. 20 reviewed items across 4 content types.
📝Interview Questions11Q&A with detailed answers and code examples→
🏭Use Cases3Real-world production patterns and scenarios→
🔥Production Issues3On-call incidents with root causes and fixes→
⭐Best Practices3Security, scaling, and day-2 operations→
11 / 11
⚡
Quick Answer
Change sets preview what CloudFormation intends to create, update, replace, or delete. Stack policies constrain dangerous updates, rollback returns a failed stack toward the prior stable state, and drift detection finds unmanaged changes made outside CloudFormation.
Detailed Answer
CloudFormation is like a building permit office. The template is the approved blueprint, a change set is the proposed renovation, a stack policy protects load-bearing walls, and drift detection finds rooms someone changed without a permit.
CloudFormation is built around declarative desired state for AWS resources. Production change control depends on reviewing the difference between template intent and deployed reality, then limiting which critical resources can be replaced.
When a stack update starts, CloudFormation builds a dependency graph from resources and references, calls AWS service APIs in order, waits for stabilization signals, and records stack events. Drift detection compares supported resource properties against current live configuration.
At scale, teams require change sets in CI, protect stateful resources with stack policies and deletion policies, and watch stack events during updates. They use drift detection after emergency console changes and before large refactors.
A non-obvious gotcha is that drift detection is not universal for every property or resource type. A clean drift report does not prove the system is identical; it proves CloudFormation found no drift in the properties it can check.
Code Example
aws cloudformation create-change-set --stack-name payments-prod --change-set-name release-2026-06-18 --template-body file://template.yaml --capabilities CAPABILITY_IAM # Creates a reviewable production change set. aws cloudformation describe-change-set --stack-name payments-prod --change-set-name release-2026-06-18 # Shows replacements, deletes, and IAM changes before execution. aws cloudformation detect-stack-drift --stack-name payments-prod # Starts drift detection to find unmanaged production edits.
💡
Interview Tip
A junior engineer typically answers with the feature name and a happy-path command, but for a senior/architect role, the interviewer is actually looking for production judgment. Explain where the CloudFormation control plane stores intent, how changes are rolled out, what telemetry proves the system is healthy, and what failure mode creates the largest blast radius. Strong answers include rollback behavior, ownership boundaries, permissions, and the exact signal you would page on.
◈ Architecture Diagram
┌──────────┐
│ Signal │
└────┬─────┘
↓
┌──────────┐
│ CloudFor │
└────┬─────┘
↓
┌──────────┐
│ Action │
└──────────┘💬 Comments
⚡
Quick Answer
CloudFormation for AWS-only, compliance-heavy environments with native service integration; Terraform for multi-cloud or multi-provider workflows with mature state management; CDK when developer teams want to use programming languages while getting CloudFormation's native AWS integration under the hood.
Detailed Answer
CloudFormation Strengths
CloudFormation is AWS's native IaC service with zero additional cost, automatic state management (no separate state file), deep integration with AWS services (often supports new services on launch day), and native support for features like StackSets (multi-account deployment), drift detection, and Change Sets. It operates within AWS's control plane, meaning IAM policies control who can deploy what. For regulated industries, CloudFormation's native integration with AWS Config, CloudTrail, and Service Catalog is a significant advantage.
Terraform Strengths
Terraform supports multi-cloud (AWS, Azure, GCP, Kubernetes) and multi-provider (Datadog, PagerDuty, GitHub) workflows with a single tool. Its plan-apply workflow is more explicit than CloudFormation's change sets. The module registry provides reusable patterns. State management via remote backends (S3, Terraform Cloud) is mature. Terraform's provider ecosystem is broader - you can manage DNS, monitoring, CI/CD, and cloud resources in one codebase.
AWS CDK Strengths
CDK lets developers write infrastructure in TypeScript, Python, Java, Go, or C# and synthesizes it to CloudFormation templates. This gives you real programming language features (loops, conditionals, type checking, IDE support) while retaining CloudFormation's native deployment engine. CDK constructs provide higher-level abstractions than raw CloudFormation, and the construct library is growing rapidly.
Decision Framework
Choose CloudFormation when you're AWS-only, need native compliance features, and your team is comfortable with YAML/JSON. Choose Terraform when you're multi-cloud, need to manage non-AWS resources, or want the plan-apply workflow. Choose CDK when your team has strong programming skills and you want CloudFormation's native benefits with better developer experience. Avoid mixing all three in one organization without clear boundaries.
Code Example
# CloudFormation - Native YAML with intrinsic functions
AWSTemplateFormatVersion: '2010-09-09'
Resources:
MyVPC:
Type: AWS::EC2::VPC
Properties:
CidrBlock: 10.0.0.0/16
EnableDnsHostnames: true
Tags:
- Key: Name
Value: !Sub '${AWS::StackName}-vpc'
# Terraform - HCL with multi-provider support
resource "aws_vpc" "main" {
cidr_block = "10.0.0.0/16"
enable_dns_hostnames = true
tags = { Name = "${var.project}-vpc" }
}
resource "datadog_monitor" "vpc_flow" {
name = "VPC Flow Log Anomaly"
type = "metric alert"
query = "avg:aws.vpc.flowlogs.rejected{vpc:${aws_vpc.main.id}}"
}
# CDK (TypeScript) - Programming language + CloudFormation engine
import * as ec2 from 'aws-cdk-lib/aws-ec2';
const vpc = new ec2.Vpc(this, 'MyVPC', {
maxAzs: 3,
natGateways: 1,
subnetConfiguration: [
{ name: 'Public', subnetType: ec2.SubnetType.PUBLIC, cidrMask: 24 },
{ name: 'Private', subnetType: ec2.SubnetType.PRIVATE_WITH_EGRESS, cidrMask: 24 },
],
});
// CDK generates ~200 lines of CloudFormation from this 10-line construct💡
Interview Tip
Don't pick a winner. Show you understand each tool's sweet spot. Mention that CDK synthesizes to CloudFormation (so it's not a completely separate tool) and that Terraform's multi-provider capability is its unique differentiator. Cost is often overlooked: CloudFormation is free, Terraform Cloud has per-resource pricing.
💬 Comments
⚡
Quick Answer
Split the monolithic stack into layered nested stacks by resource lifecycle (networking, data, compute, application). Use cross-stack references via Exports/Imports for shared values. Implement a parent stack that orchestrates deployment order, and use change sets to validate changes before applying.
Detailed Answer
Nested Stack Architecture
Nested stacks allow you to break a monolithic template into modular, reusable child templates. The parent stack references child stacks using AWS::CloudFormation::Stack resources. Each child stack has its own lifecycle and can be updated independently. Structure your hierarchy by deployment frequency and blast radius: networking (rarely changes) at the bottom, databases (occasional changes) in the middle, application services (frequent changes) at the top.
Cross-Stack References
For values shared between independent stacks (not nested), use CloudFormation Exports and Fn::ImportValue. The networking stack exports VPC ID, subnet IDs, and security group IDs. The database stack imports VPC ID and exports connection endpoints. The application stack imports both. Important limitation: you cannot delete or modify an exported value while any other stack imports it. This creates a coupling that requires careful management.
Operational Improvements
With layered stacks, a change to an ECS service only updates the application stack (minutes) instead of the entire infrastructure (45 minutes). Rollbacks are faster because they're scoped to the affected layer. Each layer can have its own update policy and rollback configuration. Use DependsOn between nested stacks to enforce deployment ordering.
Migration Strategy
Migrating from a monolithic stack to nested stacks requires careful planning. You cannot simply split resources into new stacks because CloudFormation would try to delete resources from the original stack. Instead: (1) Add DeletionPolicy: Retain to all resources, (2) Create new stacks that import existing resources using aws cloudformation import, (3) Remove resources from the original stack, (4) Verify state consistency.
Code Example
# Parent stack: master-template.yaml
AWSTemplateFormatVersion: '2010-09-09'
Description: Parent stack orchestrating infrastructure layers
Parameters:
Environment:
Type: String
AllowedValues: [dev, staging, prod]
Resources:
NetworkStack:
Type: AWS::CloudFormation::Stack
Properties:
TemplateURL: !Sub 'https://s3.amazonaws.com/cfn-templates-${AWS::AccountId}/network.yaml'
Parameters:
Environment: !Ref Environment
VpcCidr: '10.0.0.0/16'
Tags:
- Key: Layer
Value: networking
DatabaseStack:
Type: AWS::CloudFormation::Stack
DependsOn: NetworkStack
Properties:
TemplateURL: !Sub 'https://s3.amazonaws.com/cfn-templates-${AWS::AccountId}/database.yaml'
Parameters:
Environment: !Ref Environment
VpcId: !GetAtt NetworkStack.Outputs.VpcId
PrivateSubnetIds: !GetAtt NetworkStack.Outputs.PrivateSubnetIds
ApplicationStack:
Type: AWS::CloudFormation::Stack
DependsOn: DatabaseStack
Properties:
TemplateURL: !Sub 'https://s3.amazonaws.com/cfn-templates-${AWS::AccountId}/application.yaml'
Parameters:
Environment: !Ref Environment
VpcId: !GetAtt NetworkStack.Outputs.VpcId
DbEndpoint: !GetAtt DatabaseStack.Outputs.DbEndpoint
---
# network.yaml - Child stack with Exports
Outputs:
VpcId:
Value: !Ref VPC
Export:
Name: !Sub '${Environment}-VpcId'
PrivateSubnetIds:
Value: !Join [',', [!Ref PrivateSubnet1, !Ref PrivateSubnet2]]
Export:
Name: !Sub '${Environment}-PrivateSubnetIds'
---
# Cross-stack reference in independent stack
Resources:
MySecurityGroup:
Type: AWS::EC2::SecurityGroup
Properties:
VpcId: !ImportValue
Fn::Sub: '${Environment}-VpcId'
# Validate change set before applying
aws cloudformation create-change-set \
--stack-name prod-master \
--template-url https://s3.amazonaws.com/cfn-templates/master.yaml \
--change-set-name update-app-layer \
--parameters ParameterKey=Environment,ParameterValue=prod
aws cloudformation describe-change-set \
--stack-name prod-master \
--change-set-name update-app-layer \
--query 'Changes[].ResourceChange.{Action:Action,Resource:LogicalResourceId,Type:ResourceType}'💡
Interview Tip
Acknowledge the Export/Import coupling limitation - you cannot update an export while it's imported. This is a real operational pain point. Mention the import-resource strategy for migrating from monolithic to nested stacks without downtime. Always discuss change sets as a safety mechanism.
💬 Comments
⚡
Quick Answer
Create a Lambda function that handles Create, Update, and Delete lifecycle events from CloudFormation, processes the custom logic, and sends a SUCCESS or FAILED response to the pre-signed S3 URL. Reference it in the template using `Custom::ResourceType` with `ServiceToken` pointing to the Lambda ARN.
Detailed Answer
Custom Resource Lifecycle
When CloudFormation encounters a custom resource, it sends a JSON event to the Lambda function containing the RequestType (Create, Update, Delete), ResourceProperties (your custom parameters), StackId, LogicalResourceId, and a ResponseURL (pre-signed S3 URL). The Lambda must process the request and send an HTTP PUT to the ResponseURL with a JSON body containing Status (SUCCESS/FAILED), PhysicalResourceId, and any output Data. If the Lambda fails to respond, CloudFormation waits for the timeout (default 1 hour) before failing.
Critical Implementation Details
The PhysicalResourceId is crucial: if it changes between Create and Update, CloudFormation triggers a Delete of the old resource. Always return a stable physical ID unless you intend replacement. The cfn-response module (provided in the Lambda runtime) simplifies sending responses, but for production use, implement your own response logic with proper error handling. Always wrap your Lambda in a try-catch to ensure a response is sent even on errors - an unhandled exception means CloudFormation hangs for an hour.
CloudFormation Resource Provider Framework
For production custom resources, use the CloudFormation CLI (cfn init) to build a proper resource provider. This generates a schema, handler code, and test harnesses. Registered resource types appear as AWS::MyCompany::ResourceType and support drift detection, import, and all standard CloudFormation lifecycle operations. This is significantly more robust than raw Lambda-backed custom resources.
Common Use Cases
Custom resources are commonly used for: looking up AMI IDs dynamically, emptying S3 buckets before deletion, configuring third-party services (Datadog, PagerDuty), running database migrations, waiting for DNS propagation, and any operation that requires imperative logic during stack operations.
Code Example
# CloudFormation template with custom resource
Resources:
DatadogDashboardFunction:
Type: AWS::Lambda::Function
Properties:
FunctionName: cfn-datadog-dashboard
Runtime: python3.12
Handler: index.handler
Timeout: 300
Environment:
Variables:
DD_API_KEY: !Sub '{{resolve:secretsmanager:datadog-api-key}}'
DD_APP_KEY: !Sub '{{resolve:secretsmanager:datadog-app-key}}'
Code:
ZipFile: |
import json
import urllib3
import os
import traceback
http = urllib3.PoolManager()
DD_API = 'https://api.datadoghq.com/api/v1'
def send_response(event, context, status, data={}, reason=''):
body = json.dumps({
'Status': status,
'Reason': reason or f'See CloudWatch Log: {context.log_group_name}',
'PhysicalResourceId': data.get('DashboardId', event.get('PhysicalResourceId', 'none')),
'StackId': event['StackId'],
'RequestId': event['RequestId'],
'LogicalResourceId': event['LogicalResourceId'],
'Data': data
})
http.request('PUT', event['ResponseURL'], body=body,
headers={'Content-Type': 'application/json'})
def handler(event, context):
try:
props = event['ResourceProperties']
headers = {
'DD-API-KEY': os.environ['DD_API_KEY'],
'DD-APPLICATION-KEY': os.environ['DD_APP_KEY'],
'Content-Type': 'application/json'
}
if event['RequestType'] == 'Create':
resp = http.request('POST', f'{DD_API}/dashboard',
body=json.dumps({
'title': props['Title'],
'layout_type': 'ordered',
'widgets': json.loads(props['WidgetsJson'])
}), headers=headers)
dashboard = json.loads(resp.data)
send_response(event, context, 'SUCCESS',
{'DashboardId': dashboard['id'], 'DashboardUrl': dashboard['url']})
elif event['RequestType'] == 'Update':
dashboard_id = event['PhysicalResourceId']
http.request('PUT', f'{DD_API}/dashboard/{dashboard_id}',
body=json.dumps({
'title': props['Title'],
'layout_type': 'ordered',
'widgets': json.loads(props['WidgetsJson'])
}), headers=headers)
send_response(event, context, 'SUCCESS',
{'DashboardId': dashboard_id})
elif event['RequestType'] == 'Delete':
dashboard_id = event['PhysicalResourceId']
if dashboard_id != 'none':
http.request('DELETE', f'{DD_API}/dashboard/{dashboard_id}',
headers=headers)
send_response(event, context, 'SUCCESS')
except Exception as e:
traceback.print_exc()
send_response(event, context, 'FAILED', reason=str(e))
DatadogDashboard:
Type: Custom::DatadogDashboard
Properties:
ServiceToken: !GetAtt DatadogDashboardFunction.Arn
Title: !Sub '${AWS::StackName} Service Dashboard'
WidgetsJson: !Sub |
[{"definition":{"type":"timeseries","requests":[{"q":"avg:aws.ecs.cpuutilization{service:${ServiceName}}"}],"title":"CPU"}}]💡
Interview Tip
The number one production issue with custom resources is forgetting to send a response on error, causing CloudFormation to hang for an hour. Always wrap in try-catch. Mention PhysicalResourceId stability - if it changes, CloudFormation triggers a delete of the old resource.
💬 Comments
⚡
Quick Answer
Use `aws cloudformation detect-stack-drift` to identify drifted resources, `describe-stack-resource-drifts` to see exact property changes, then remediate by either updating the template to match actual state or re-deploying to restore desired state, depending on whether the drift was intentional.
Detailed Answer
Drift Detection Mechanics
CloudFormation drift detection compares the actual resource configuration (from AWS API calls) against the expected configuration (from the last successful stack operation). It reports three statuses: IN_SYNC, MODIFIED (properties changed), and DELETED (resource removed). Not all resource types support drift detection - check the AWS documentation for supported types. Run detection via CLI, console, or schedule it via EventBridge + Lambda for continuous monitoring.
Investigation Workflow
When drift is detected, examine each drifted resource's property differences. The describe-stack-resource-drifts API returns the expected value, actual value, and difference type (ADD, REMOVE, NOT_EQUAL) for each property. Common drift sources: security teams adding tags, auto-scaling modifying instance counts, manual hotfixes during incidents, and AWS service-linked role modifications. Categorize drift as intentional (authorized changes) or unintentional (configuration drift).
Remediation Strategies
For unintentional drift (someone manually changed a security group): update the stack to reassert the desired state. Create a change set to verify what will happen, then execute it. For intentional drift (security team added a required tag): update the template to include the change, then deploy to bring the template in sync with reality. For complex drift (resource was replaced): you may need to import the new resource into CloudFormation using resource import.
Prevention
Implement preventive controls: SCP policies that deny direct resource modification for CloudFormation-managed resources (using aws:cloudformation:stack-id condition key), AWS Config rules that alert on changes to tagged resources, and IAM policies that restrict write access to production resources. Establish a culture where all changes go through IaC, even during incidents (use fast-track change sets).
Code Example
# Detect drift on a stack
aws cloudformation detect-stack-drift \
--stack-name production-networking
# Check drift detection status
aws cloudformation describe-stack-drift-detection-status \
--stack-drift-detection-id a]1b2c3d4-5678-90ab-cdef
# Get detailed drift results
aws cloudformation describe-stack-resource-drifts \
--stack-name production-networking \
--stack-resource-drift-status-filters MODIFIED DELETED \
--query 'StackResourceDrifts[].{Resource:LogicalResourceId,Type:ResourceType,Status:StackResourceDriftStatus,Diff:PropertyDifferences}' \
--output table
# Example drift output:
# Resource: WebServerSG
# Type: AWS::EC2::SecurityGroup
# PropertyDifference:
# PropertyPath: /SecurityGroupIngress/1
# ExpectedValue: (none)
# ActualValue: {"CidrIp":"203.0.113.50/32","FromPort":22,"ToPort":22}
# DifferenceType: ADD
# -> Someone added SSH access manually!
# Remediation Option 1: Re-deploy to remove unauthorized change
aws cloudformation create-change-set \
--stack-name production-networking \
--template-url https://s3.amazonaws.com/templates/network.yaml \
--change-set-name remediate-drift \
--parameters ParameterKey=Environment,UsePreviousValue=true
aws cloudformation execute-change-set \
--stack-name production-networking \
--change-set-name remediate-drift
# Remediation Option 2: Import manually-created resource
aws cloudformation create-change-set \
--stack-name production-networking \
--change-set-name import-new-sg \
--change-set-type IMPORT \
--resources-to-import "[{\"ResourceType\":\"AWS::EC2::SecurityGroup\",\"LogicalResourceId\":\"NewSecurityGroup\",\"ResourceIdentifier\":{\"GroupId\":\"sg-0abc123def456\"}}]" \
--template-body file://updated-template.yaml
# Prevention: SCP to block direct modification of CFN-managed resources
{
"Version": "2012-10-17",
"Statement": [{
"Sid": "DenyDirectModification",
"Effect": "Deny",
"Action": ["ec2:AuthorizeSecurityGroupIngress", "ec2:RevokeSecurityGroupIngress"],
"Resource": "*",
"Condition": {
"StringNotEquals": {
"aws:CalledVia": ["cloudformation.amazonaws.com"]
},
"Null": {
"aws:ResourceTag/aws:cloudformation:stack-id": "false"
}
}
}]
}
# Automated drift detection via EventBridge (runs daily)
aws events put-rule \
--name daily-drift-detection \
--schedule-expression 'rate(1 day)'
# Lambda target runs detect-stack-drift on all production stacks💡
Interview Tip
The SCP-based prevention strategy is a strong differentiator in interviews. Using `aws:CalledVia` condition key to only allow changes through CloudFormation shows deep AWS IAM knowledge. Mention that not all resource types support drift detection - this shows practical experience.
💬 Comments
⚡
Quick Answer
Use CloudFormation StackSets with service-managed permissions (AWS Organizations integration), deploy to organizational units (OUs), configure concurrent deployment settings, use auto-deployment for new accounts, and implement proper failure tolerance and rollback thresholds.
Detailed Answer
StackSets Architecture
StackSets extend CloudFormation to deploy stacks across multiple AWS accounts and regions from a single management account. A StackSet is a template + configuration, and stack instances are the actual deployments in target accounts/regions. You manage one template; StackSets handles the fan-out. There are two permission models: self-managed (requires IAM roles in each target account) and service-managed (uses AWS Organizations, recommended for scale).
Service-Managed vs Self-Managed
Service-managed StackSets integrate with AWS Organizations, allowing you to target OUs instead of individual account IDs. When a new account is added to the OU, StackSets automatically deploys the stack instance (auto-deployment). This is essential for landing zones where new accounts must be pre-configured. Self-managed StackSets require pre-created IAM roles (AWSCloudFormationStackSetAdministrationRole in management, AWSCloudFormationStackSetExecutionRole in targets) and explicit account ID lists.
Deployment Strategy
Configure MaxConcurrentCount or MaxConcurrentPercentage to control how many accounts deploy simultaneously. Set FailureToleranceCount to define how many account failures are acceptable before the entire operation stops. For critical infrastructure, deploy to one region first (canary), verify, then deploy to remaining regions. Use StackSet parameters with account-level overrides for region-specific configuration.
Operational Challenges
Stack instances can drift independently. Use StackSets drift detection to check all instances at once. Template updates propagate to all instances, but failing instances require manual investigation. Parameter overrides per account/region add complexity. For large-scale deployments (100+ accounts), StackSets operations can take hours - plan maintenance windows accordingly.
Code Example
# Create a service-managed StackSet targeting an OU
aws cloudformation create-stack-set \
--stack-set-name security-baseline \
--template-url https://s3.amazonaws.com/cfn-templates/security-baseline.yaml \
--permission-model SERVICE_MANAGED \
--auto-deployment Enabled=true,RetainStacksOnAccountRemoval=false \
--capabilities CAPABILITY_NAMED_IAM \
--parameters \
ParameterKey=GuardDutyEnabled,ParameterValue=true \
ParameterKey=CloudTrailBucketName,ParameterValue=org-cloudtrail-central \
--managed-execution Active=true
# Deploy to OUs in multiple regions
aws cloudformation create-stack-instances \
--stack-set-name security-baseline \
--deployment-targets OrganizationalUnitIds='["ou-abc123","ou-def456"]' \
--regions us-east-1 us-west-2 eu-west-1 ap-southeast-1 \
--operation-preferences \
RegionConcurrencyType=PARALLEL,\
MaxConcurrentPercentage=25,\
FailureTolerancePercentage=10
# Override parameters for specific accounts
aws cloudformation create-stack-instances \
--stack-set-name security-baseline \
--deployment-targets Accounts='["111111111111"]' \
--regions us-east-1 \
--parameter-overrides ParameterKey=GuardDutyEnabled,ParameterValue=false
# Check deployment status
aws cloudformation describe-stack-set-operation \
--stack-set-name security-baseline \
--operation-id op-1234567890 \
--query '{Status:Status,Action:Action,CreationTimestamp:CreationTimestamp}'
# List all stack instances and their status
aws cloudformation list-stack-instances \
--stack-set-name security-baseline \
--query 'Summaries[].{Account:Account,Region:Region,Status:StackInstanceStatus.DetailedStatus}' \
--output table
# Detect drift across all instances
aws cloudformation detect-stack-set-drift \
--stack-set-name security-baseline
# Template excerpt: security-baseline.yaml
AWSTemplateFormatVersion: '2010-09-09'
Description: Security baseline for all accounts
Conditions:
EnableGuardDuty: !Equals [!Ref GuardDutyEnabled, 'true']
Resources:
CloudTrail:
Type: AWS::CloudTrail::Trail
Properties:
TrailName: org-trail
S3BucketName: !Ref CloudTrailBucketName
IsMultiRegionTrail: true
EnableLogFileValidation: true
IsLogging: true
GuardDutyDetector:
Type: AWS::GuardDuty::Detector
Condition: EnableGuardDuty
Properties:
Enable: true
FindingPublishingFrequency: FIFTEEN_MINUTES
ConfigRecorder:
Type: AWS::Config::ConfigurationRecorder
Properties:
RecordingGroup:
AllSupported: true
IncludeGlobalResourceTypes: true
PasswordPolicy:
Type: AWS::IAM::AccountPasswordPolicy
Properties:
MinimumPasswordLength: 14
RequireSymbols: true
RequireNumbers: true
RequireUppercaseCharacters: true
RequireLowercaseCharacters: true
MaxPasswordAge: 90
PasswordReusePrevention: 24💡
Interview Tip
Emphasize service-managed permissions with auto-deployment - this is the modern approach for landing zones. Discuss failure tolerance settings as a production safety mechanism. Mention the canary approach: deploy to one region first, verify, then fan out.
💬 Comments
⚡
Quick Answer
CloudFormation Guard (cfn-guard) is a policy-as-code tool that validates CloudFormation templates against rules written in a domain-specific language. Integrate it into CI/CD pipelines to block non-compliant templates before deployment, and use it with CloudFormation Hooks for server-side enforcement.
Detailed Answer
CloudFormation Guard Overview
Cfn-guard is an open-source CLI tool that evaluates CloudFormation (and other JSON/YAML) templates against policy rules. Rules are written in a purpose-built DSL that reads naturally: AWS::S3::Bucket { Properties.BucketEncryption EXISTS }. It runs offline (no AWS API calls needed), making it fast and suitable for CI/CD pipelines and local development. Guard 2.x introduced a significantly improved rule language with support for custom messages, cross-resource checks, and when-conditions.
Rule Writing
Guard rules use a declarative syntax that specifies resource type filters and property assertions. You can check for existence, specific values, ranges, regex patterns, and nested properties. Rules can be grouped into named rule sets with descriptions. The when clause adds conditional logic: only check multi-AZ on RDS if the environment parameter is 'prod'. Custom error messages make violations actionable for developers.
CloudFormation Hooks
For server-side enforcement, register Guard rules as CloudFormation Hooks. Hooks run automatically during stack operations and can block non-compliant resource creation/modification at the CloudFormation level, regardless of whether the deployment comes from CLI, console, or CI/CD. This provides defense-in-depth: even if a developer bypasses CI/CD policy checks, the Hook catches the violation.
Integration Patterns
Run cfn-guard in CI/CD as a pre-deployment gate: cfn-guard validate -d template.yaml -r rules/. Store rules in a central Git repository shared across teams. Use the cfn-guard test command to unit test your rules against sample templates. Combine with AWS Config conformance packs for runtime compliance monitoring.
Code Example
# Install cfn-guard
curl --proto '=https' --tlsv1.2 -sSf https://raw.githubusercontent.com/aws-cloudformation/cloudformation-guard/main/install-guard.sh | sh
# rules/security.guard - Policy rules file
# Rule 1: S3 buckets must have encryption
rule s3_encryption_required when resourceType == "AWS::S3::Bucket" {
Properties.BucketEncryption EXISTS
Properties.BucketEncryption.ServerSideEncryptionConfiguration[*] {
ServerSideEncryptionByDefault.SSEAlgorithm IN ["aws:kms", "AES256"]
}
<<
Violation: S3 bucket must have server-side encryption enabled with KMS or AES256.
Fix: Add BucketEncryption with SSEAlgorithm set to 'aws:kms' or 'AES256'.
>>
}
# Rule 2: No public security group ingress on sensitive ports
rule no_public_ssh_rdp when resourceType == "AWS::EC2::SecurityGroup" {
Properties.SecurityGroupIngress[*] {
when FromPort == 22 OR FromPort == 3389 {
CidrIp != "0.0.0.0/0"
CidrIpv6 != "::/0"
<<
Violation: SSH (22) and RDP (3389) must not be open to 0.0.0.0/0.
Fix: Restrict CidrIp to your corporate CIDR range.
>>
}
}
}
# Rule 3: RDS must be encrypted and multi-AZ in production
rule rds_production_requirements when resourceType == "AWS::RDS::DBInstance" {
Properties.StorageEncrypted == true
Properties.DeletionProtection == true
when Parameters.Environment.Default == "prod" {
Properties.MultiAZ == true
Properties.BackupRetentionPeriod >= 7
}
}
# Rule 4: All resources must have required tags
rule required_tags {
Resources[*] {
when Type IN ["AWS::EC2::Instance", "AWS::RDS::DBInstance", "AWS::S3::Bucket"] {
Properties.Tags[*] {
some Key == "Environment"
some Key == "Owner"
some Key == "CostCenter"
}
}
}
}
# Validate template against rules
cfn-guard validate \
--data template.yaml \
--rules rules/security.guard \
--show-summary all \
--output-format json
# Unit test rules against sample templates
# tests/security_tests.yaml
- name: S3 bucket without encryption should FAIL
input:
Resources:
BadBucket:
Type: AWS::S3::Bucket
Properties:
BucketName: my-bucket
expectations:
rules:
s3_encryption_required: FAIL
cfn-guard test --rules-file rules/security.guard --test-data tests/
# CI/CD integration (GitHub Actions)
# - name: Validate CloudFormation
# run: |
# cfn-guard validate -d templates/ -r rules/ --show-summary fail
# if [ $? -ne 0 ]; then echo "Policy violations found!"; exit 1; fi💡
Interview Tip
Distinguish between cfn-guard (client-side, CI/CD) and CloudFormation Hooks (server-side, defense-in-depth). Show that you understand both layers are needed: CI/CD catches 95% but Hooks catch the rest. Mention unit testing rules with `cfn-guard test` to show you treat policies as code.
💬 Comments
⚡
Quick Answer
Identify the resource causing the rollback failure from stack events, fix the underlying issue (often a resource that was manually deleted or modified), then use `continue-update-rollback` with `--resources-to-skip` to skip the problematic resource and complete the rollback. As a last resort, delete the stack with `--retain-resources`.
Detailed Answer
Understanding UPDATE_ROLLBACK_FAILED
This state occurs when CloudFormation tries to roll back a failed update but encounters an error during the rollback itself. Common causes: a resource was manually deleted outside CloudFormation and can't be restored, an IAM permission was removed, a resource dependency changed, or the rollback requires creating a resource that conflicts with an existing one. The stack is now stuck and no further updates are possible until the rollback completes.
Diagnosis Steps
Examine stack events to find the exact resource and error message causing the rollback failure. Look for events with UPDATE_ROLLBACK_FAILED or DELETE_FAILED status reason. Common error messages include 'Resource X does not exist', 'Access Denied', or 'Limit exceeded'. Cross-reference with CloudTrail to see if someone modified resources outside CloudFormation.
Recovery with continue-update-rollback
Once you understand which resources are blocking the rollback, use continue-update-rollback with --resources-to-skip to tell CloudFormation to ignore those resources and complete the rollback. Skipped resources will be removed from CloudFormation's state management (they become unmanaged). After the rollback completes, you can import them back into the stack or manage them separately.
Prevention
Protect critical resources with DeletionPolicy: Retain and UpdateReplacePolicy: Retain. Use stack policies to prevent accidental updates to sensitive resources. Implement SCP policies that prevent manual modification of CloudFormation-managed resources. Always test changes in a non-production environment first.
Code Example
# Step 1: Diagnose - find the failing resource
aws cloudformation describe-stack-events \
--stack-name production-app \
--query 'StackEvents[?ResourceStatus==`UPDATE_ROLLBACK_FAILED` || ResourceStatus==`DELETE_FAILED`].{Time:Timestamp,Resource:LogicalResourceId,Type:ResourceType,Status:ResourceStatus,Reason:ResourceStatusReason}' \
--output table
# Example output:
# Resource: ApiGatewayDeployment
# Reason: "The REST API doesn't exist (Service: AmazonApiGateway; Status Code: 404)"
# -> Someone deleted the API Gateway outside CloudFormation
# Step 2: Attempt to fix the root cause
# If the resource was manually deleted, sometimes you can recreate it
# If it's an IAM issue, fix the permissions
# Step 3: Continue rollback, skipping the problematic resource
aws cloudformation continue-update-rollback \
--stack-name production-app \
--resources-to-skip ApiGatewayDeployment ApiGatewayStage
# Step 4: Verify stack is back to UPDATE_ROLLBACK_COMPLETE
aws cloudformation describe-stacks \
--stack-name production-app \
--query 'Stacks[0].{Status:StackStatus,Reason:StackStatusReason}'
# Step 5: Now you can update the stack again
# Fix the template to handle the missing resources
aws cloudformation update-stack \
--stack-name production-app \
--template-url https://s3.amazonaws.com/templates/app-fixed.yaml \
--parameters ParameterKey=Environment,UsePreviousValue=true
# Prevention: Stack policy to protect critical resources
aws cloudformation set-stack-policy \
--stack-name production-app \
--stack-policy-body '{
"Statement": [
{
"Effect": "Deny",
"Action": "Update:Replace",
"Principal": "*",
"Resource": "LogicalResourceId/ProductionDatabase"
},
{
"Effect": "Deny",
"Action": "Update:Delete",
"Principal": "*",
"Resource": "LogicalResourceId/ProductionDatabase"
},
{
"Effect": "Allow",
"Action": "Update:*",
"Principal": "*",
"Resource": "*"
}
]
}'
# Prevention: DeletionPolicy in template
Resources:
ProductionDatabase:
Type: AWS::RDS::DBInstance
DeletionPolicy: Retain
UpdateReplacePolicy: Retain
Properties:
DBInstanceClass: db.r5.2xlarge
Engine: postgres
MultiAZ: true
DeletionProtection: true
# Nuclear option: Delete stack but retain critical resources
aws cloudformation delete-stack \
--stack-name production-app \
--retain-resources ProductionDatabase ProductionVPC💡
Interview Tip
This is a very common production scenario. Walk through the diagnostic workflow step by step: events, identify the failing resource, understand why, fix or skip. Emphasize `--resources-to-skip` as the primary recovery tool. Mention that skipped resources become unmanaged - this has implications.
💬 Comments
⚡
Quick Answer
Create a CloudFormation macro backed by a Lambda function that processes the template fragment during deployment. The Lambda receives the template, modifies it programmatically (adding tags, injecting log groups, applying security defaults), and returns the transformed template. Reference it with `Transform` in the template.
Detailed Answer
CloudFormation Macros
Macros are template-processing functions that run during CloudFormation's template processing phase. When CloudFormation encounters a Transform reference, it sends the template (or a fragment) to the associated Lambda function. The Lambda modifies the template JSON and returns the result. CloudFormation then processes the transformed template as if you wrote it that way. The most well-known macro is AWS::Serverless-2016-10-31 (SAM transform).
Macro Types
Template-level macros process the entire template and are referenced in the Transform section at the top level. Snippet-level macros process specific sections and are invoked via Fn::Transform within a resource definition. Template-level macros are ideal for organization-wide standards (inject tags, logging), while snippet-level macros are useful for reusable resource patterns (generate a standard monitoring stack for any service).
Implementation
The Lambda receives an event with templateParameterValues, fragment (the template section to process), requestId, and params (macro-specific parameters). It must return requestId, status (success/failure), and fragment (the modified template). The Lambda can add resources, modify properties, inject conditions, and transform the template in any way. It runs during the change set creation phase, so you can see the transformed result before deploying.
Use Cases and Limitations
Common use cases: inject standard tags across all resources, add CloudWatch log groups for every Lambda, enforce encryption on all storage resources, add WAF rules to all API Gateways, and create standard alarm sets. Limitations: macros add complexity and make templates harder to read (the deployed template differs from the source). Debugging macro output requires examining the processed template. Macro Lambdas must be deployed before they can be used in templates.
Code Example
# Step 1: Lambda function for the macro
# macro-lambda/index.py
import copy
def handler(event, context):
fragment = event['fragment']
status = 'success'
try:
# Inject standard tags into all taggable resources
standard_tags = [
{'Key': 'ManagedBy', 'Value': 'CloudFormation'},
{'Key': 'Environment', 'Value': event['templateParameterValues'].get('Environment', 'dev')},
{'Key': 'CostCenter', 'Value': event['templateParameterValues'].get('CostCenter', 'engineering')},
]
taggable_types = [
'AWS::EC2::Instance', 'AWS::EC2::VPC', 'AWS::EC2::Subnet',
'AWS::S3::Bucket', 'AWS::RDS::DBInstance', 'AWS::ECS::Service',
'AWS::Lambda::Function', 'AWS::ECS::Cluster',
]
for resource_name, resource in fragment.get('Resources', {}).items():
if resource['Type'] in taggable_types:
existing_tags = resource.get('Properties', {}).get('Tags', [])
existing_keys = {t['Key'] for t in existing_tags}
for tag in standard_tags:
if tag['Key'] not in existing_keys:
existing_tags.append(tag)
resource.setdefault('Properties', {})['Tags'] = existing_tags
# Inject CloudWatch log group for every Lambda
if resource['Type'] == 'AWS::Lambda::Function':
func_name = resource['Properties'].get('FunctionName', resource_name)
log_group_name = f'{resource_name}LogGroup'
fragment['Resources'][log_group_name] = {
'Type': 'AWS::Logs::LogGroup',
'Properties': {
'LogGroupName': {'Fn::Sub': f'/aws/lambda/${{{resource_name}}}'},
'RetentionInDays': 30,
}
}
except Exception as e:
status = 'failure'
fragment = str(e)
return {
'requestId': event['requestId'],
'status': status,
'fragment': fragment,
}
# Step 2: Deploy the macro
# macro-template.yaml
Resources:
MacroFunction:
Type: AWS::Lambda::Function
Properties:
Runtime: python3.12
Handler: index.handler
Code:
S3Bucket: cfn-macros
S3Key: company-standards.zip
Timeout: 30
CompanyStandardsMacro:
Type: AWS::CloudFormation::Macro
Properties:
Name: CompanyStandards
FunctionName: !GetAtt MacroFunction.Arn
# Step 3: Use the macro in application templates
AWSTemplateFormatVersion: '2010-09-09'
Transform: CompanyStandards # <-- Macro invocation
Parameters:
Environment:
Type: String
Default: prod
Resources:
MyFunction:
Type: AWS::Lambda::Function
Properties:
Runtime: python3.12
Handler: app.handler
# Tags and LogGroup will be injected by macro
# View processed template (after macro transformation)
aws cloudformation get-template \
--stack-name my-stack \
--template-stage Processed💡
Interview Tip
Macros are powerful but can be a debugging nightmare if overused. Position them as a governance tool for organization-wide standards (tags, logging), not as a general-purpose template generation system. Mention that you can view the processed template to debug macro output.
💬 Comments
⚡
Quick Answer
Set `DeletionPolicy: Snapshot` (creates a final snapshot before deletion), `UpdateReplacePolicy: Snapshot` (snapshots before replacement during updates), enable stack termination protection, use stack policies to deny Replace/Delete on the database resource, and store the master password in Secrets Manager with dynamic references.
Detailed Answer
DeletionPolicy
DeletionPolicy controls what happens to a resource when it's removed from the template or the stack is deleted. Three options: Delete (default, destroys the resource), Retain (keeps the resource but removes it from CloudFormation management), and Snapshot (creates a final snapshot before deletion, only supported by RDS, EBS, ElastiCache, Neptune, Redshift). For production databases, always use Snapshot at minimum, or Retain if you never want CloudFormation to delete the instance.
UpdateReplacePolicy
Some updates require CloudFormation to replace the resource (create new, delete old). For example, changing an RDS instance's engine version might trigger replacement. UpdateReplacePolicy controls what happens to the OLD resource during replacement. Setting it to Snapshot creates a backup before the old instance is deleted. Without this, an update that triggers replacement silently deletes your production database. This is one of the most dangerous CloudFormation behaviors.
Defense in Depth
Layer multiple protections: (1) DeletionPolicy on the resource, (2) UpdateReplacePolicy on the resource, (3) Stack termination protection on the stack (prevents delete-stack), (4) Stack policies that deny Replace/Delete operations on the database, (5) RDS DeletionProtection at the AWS API level, (6) IAM/SCP policies restricting who can disable these protections. No single layer is sufficient - each protects against different failure modes.
Secrets Management
Never hardcode database passwords in CloudFormation templates. Use dynamic references to Secrets Manager: {{resolve:secretsmanager:prod/db-password:SecretString:password}}. CloudFormation resolves this at deploy time and never stores the plaintext value in the template or state. Combine with Secrets Manager rotation for automatic password rotation.
Code Example
# Production RDS with comprehensive protection
AWSTemplateFormatVersion: '2010-09-09'
Resources:
# Master password in Secrets Manager
DBMasterSecret:
Type: AWS::SecretsManager::Secret
Properties:
Name: !Sub '${AWS::StackName}/db-master-password'
GenerateSecretString:
SecretStringTemplate: '{"username": "admin"}'
GenerateStringKey: password
PasswordLength: 32
ExcludeCharacters: '"@/\\'
ProductionDatabase:
Type: AWS::RDS::DBInstance
DeletionPolicy: Snapshot # Snapshot before delete
UpdateReplacePolicy: Snapshot # Snapshot before replacement
Properties:
DBInstanceIdentifier: !Sub '${AWS::StackName}-db'
Engine: postgres
EngineVersion: '15.4'
DBInstanceClass: db.r6g.2xlarge
AllocatedStorage: 500
StorageType: gp3
StorageEncrypted: true
KmsKeyId: !Ref DatabaseKmsKey
MultiAZ: true
DeletionProtection: true # AWS-level protection
MasterUsername: !Sub '{{resolve:secretsmanager:${DBMasterSecret}:SecretString:username}}'
MasterUserPassword: !Sub '{{resolve:secretsmanager:${DBMasterSecret}:SecretString:password}}'
BackupRetentionPeriod: 35
PreferredBackupWindow: '03:00-04:00'
CopyTagsToSnapshot: true
EnablePerformanceInsights: true
MonitoringInterval: 60
MonitoringRoleArn: !GetAtt RDSMonitoringRole.Arn
Tags:
- Key: Environment
Value: production
- Key: BackupSchedule
Value: daily
# Secrets Manager rotation
DBSecretRotation:
Type: AWS::SecretsManager::RotationSchedule
Properties:
SecretId: !Ref DBMasterSecret
RotationRules:
AutomaticallyAfterDays: 30
# Enable stack termination protection
# aws cloudformation update-termination-protection \
# --enable-termination-protection \
# --stack-name production-database
# Stack policy: prevent database replacement or deletion
# aws cloudformation set-stack-policy --stack-name production-database --stack-policy-body
{
"Statement": [
{
"Effect": "Deny",
"Action": ["Update:Replace", "Update:Delete"],
"Principal": "*",
"Resource": "LogicalResourceId/ProductionDatabase",
"Condition": {
"StringEquals": {
"ResourceType": ["AWS::RDS::DBInstance"]
}
}
},
{
"Effect": "Allow",
"Action": "Update:Modify",
"Principal": "*",
"Resource": "LogicalResourceId/ProductionDatabase"
},
{
"Effect": "Allow",
"Action": "Update:*",
"Principal": "*",
"Resource": "*"
}
]
}
# Verify all protections are in place
aws cloudformation describe-stacks --stack-name production-database \
--query 'Stacks[0].EnableTerminationProtection'
aws rds describe-db-instances --db-instance-identifier production-database-db \
--query 'DBInstances[0].DeletionProtection'💡
Interview Tip
UpdateReplacePolicy is the one most candidates forget. Explain the scenario: you change an RDS property that triggers replacement, CloudFormation creates a new instance, and deletes the old one WITH YOUR DATA. UpdateReplacePolicy: Snapshot prevents this disaster. Layer all five protection mechanisms for defense in depth.
💬 Comments
⚡
Quick Answer
Implement a layered architecture with shared template modules in S3, Service Catalog for governed self-service, StackSets for cross-account baselines, CI/CD pipelines per stack with change set reviews, and CloudFormation Registry for custom resource types and modules.
Detailed Answer
Template Modularization
Create a central S3 bucket (versioned, encrypted) containing reusable nested stack templates: VPC-with-flow-logs, ECS-cluster-with-capacity-providers, RDS-with-monitoring, Lambda-with-tracing. Version templates using S3 object versioning and reference specific versions in parent stacks. This eliminates copy-paste duplication and ensures security standards are baked into shared templates. Use CloudFormation modules (registered in the CloudFormation Registry) for smaller reusable patterns that don't warrant a full nested stack.
AWS Service Catalog
For self-service provisioning, publish CloudFormation products in Service Catalog. Each product is a governed template with IAM constraints, parameter validation, and approved configurations. Application teams launch products from the catalog without needing CloudFormation expertise or broad AWS permissions. Products can be shared across accounts via portfolio sharing. This provides the guardrails organizations need at scale.
CI/CD Pipeline Architecture
Implement GitOps for CloudFormation: templates live in Git, PRs trigger linting (cfn-lint), policy validation (cfn-guard), and change set creation. Reviewers see the change set diff in the PR. Merging to main triggers change set execution. Use CodePipeline with manual approval gates for production stacks. Implement drift detection on a schedule and alert when stacks drift from their templates.
Performance Optimization
For large stacks, split by lifecycle and blast radius. Use concurrent stack deployments in CI/CD where stacks are independent. Implement resource import to bring existing resources under CloudFormation without recreating them. Use CloudFormation hooks for pre-create/pre-update validation. Tag all stacks consistently for cost attribution and operational management.
Code Example
# 1. CloudFormation Module (registered in Registry)
# fragments/standard-vpc/fragments/fragment.json
{
"AWSTemplateFormatVersion": "2010-09-09",
"Description": "Standard VPC module with flow logs and NAT",
"Parameters": {
"Environment": { "Type": "String" },
"CidrBlock": { "Type": "String", "Default": "10.0.0.0/16" }
},
"Resources": {
"VPC": {
"Type": "AWS::EC2::VPC",
"Properties": {
"CidrBlock": { "Ref": "CidrBlock" },
"EnableDnsHostnames": true
}
},
"FlowLog": {
"Type": "AWS::EC2::FlowLog",
"Properties": {
"ResourceId": { "Ref": "VPC" },
"ResourceType": "VPC",
"TrafficType": "ALL",
"LogDestinationType": "cloud-watch-logs"
}
}
}
}
# Register the module
cfn submit --dry-run
cfn submit
# Use registered module in templates
Resources:
NetworkLayer:
Type: Acme::Network::VPC::MODULE
Properties:
Environment: prod
CidrBlock: 10.0.0.0/16
# 2. Service Catalog Product
aws servicecatalog create-product \
--name "Standard ECS Service" \
--owner "Platform Team" \
--product-type CLOUD_FORMATION_TEMPLATE \
--provisioning-artifact-parameters \
'{"Name":"v2.1","Info":{"LoadTemplateFromURL":"https://s3.amazonaws.com/cfn-templates/ecs-service-v2.1.yaml"},"Type":"CLOUD_FORMATION_TEMPLATE"}'
# Add launch constraint (IAM role for provisioning)
aws servicecatalog create-constraint \
--portfolio-id port-abc123 \
--product-id prod-def456 \
--type LAUNCH \
--parameters '{"RoleArn":"arn:aws:iam::123456789:role/ServiceCatalogLaunchRole"}'
# 3. CI/CD Pipeline (buildspec.yml)
version: 0.2
phases:
install:
commands:
- pip install cfn-lint cfn-guard
validate:
commands:
- cfn-lint templates/*.yaml
- cfn-guard validate -d templates/ -r policies/
deploy:
commands:
- |
CHANGE_SET=$(aws cloudformation create-change-set \
--stack-name $STACK_NAME \
--template-url $TEMPLATE_URL \
--change-set-name deploy-$(date +%s) \
--query 'Id' --output text)
aws cloudformation wait change-set-create-complete \
--change-set-name $CHANGE_SET
aws cloudformation execute-change-set \
--change-set-name $CHANGE_SET
aws cloudformation wait stack-update-complete \
--stack-name $STACK_NAME
# 4. Drift detection across all stacks
#!/bin/bash
for stack in $(aws cloudformation list-stacks \
--stack-status-filter CREATE_COMPLETE UPDATE_COMPLETE \
--query 'StackSummaries[].StackName' --output text); do
echo "Checking drift: $stack"
aws cloudformation detect-stack-drift --stack-name $stack
done💡
Interview Tip
Focus on the organizational aspects: Service Catalog for governed self-service, modules for reuse, StackSets for baselines. Technical scaling (nested stacks, CI/CD) is expected. The differentiator is showing how CloudFormation integrates with AWS governance services (Service Catalog, Config, Organizations) for enterprise-scale management.
💬 Comments