Everything for Pulumi in one place — pick a section below. 20 reviewed items across 4 content types.
Quick Answer
A Pulumi stack is an isolated instance of a program with its own state, configuration, secrets, and outputs. Stack references create contracts between stacks, while refresh reconciles Pulumi state with real provider state so plans are not based on stale assumptions.
Detailed Answer
Think of each stack as a branch office ledger. The network office can publish official subnet IDs, and the payments office can read them, but neither office should edit the other ledger by hand.
Pulumi uses stacks so the same infrastructure program can safely represent dev, staging, and production, or separate ownership domains. This design lets teams keep code reuse while preserving state isolation and access control.
During an update, Pulumi loads the program, reads stack config and secrets, evaluates resources, compares desired resources to state, asks providers for previews, and records resulting changes. Stack references expose selected outputs without merging ownership.
Production teams use stack policies, protected resources, explicit providers, CI previews, and scheduled refreshes. They monitor failed updates, pending operations, unexpected replacements, and drift in resources that are often modified manually during emergencies.
The tricky edge case is letting stack references become hidden global variables. If many application stacks read mutable network outputs, a network rename or replacement can cause broad downstream churn unless outputs are versioned and deprecation is planned.
Code Example
pulumi stack select org/payments/prod # Selects the production state boundary before previewing changes. pulumi refresh --yes # Reconciles Pulumi state with provider reality before planning a risky update. pulumi preview --diff # Shows exact proposed replacements, updates, and deletes for review. pulumi up --yes # Applies the reviewed infrastructure change.
Interview Tip
A junior engineer typically answers with the feature name and a happy-path command, but for a senior/architect role, the interviewer is actually looking for production judgment. Explain where the Pulumi control plane stores intent, how changes are rolled out, what telemetry proves the system is healthy, and what failure mode creates the largest blast radius. Strong answers include rollback behavior, ownership boundaries, permissions, and the exact signal you would page on.
◈ Architecture Diagram
┌──────────┐
│ Signal │
└────┬─────┘
↓
┌──────────┐
│ Pulumi │
└────┬─────┘
↓
┌──────────┐
│ Action │
└──────────┘💬 Comments
Quick Answer
Pulumi uses general-purpose languages (TypeScript, Python, Go) enabling loops, conditionals, and type safety natively, while Terraform uses declarative HCL. Pulumi excels when teams have strong software engineering practices; Terraform excels when you want strict declarative guardrails and a broader module ecosystem.
Detailed Answer
Pulumi allows you to write infrastructure code in TypeScript, Python, Go, C#, or Java. This means you get IDE autocompletion, type checking, refactoring tools, and the ability to use any library from the language ecosystem. Terraform's HCL is purpose-built for infrastructure and enforces a declarative model where you describe the desired end state without imperative logic. The trade-off is expressiveness vs. guardrails: Pulumi lets you write for loops and if statements naturally, while Terraform requires count, for_each, and ternary expressions that can become unwieldy for complex logic.
Both tools track state. Terraform uses state files stored in backends (S3, GCS, Terraform Cloud). Pulumi uses a similar concept but defaults to the Pulumi Cloud service as its backend, which provides encryption, history, and RBAC out of the box. You can also self-host Pulumi state in S3 or Azure Blob. One critical difference: Pulumi encrypts secrets in state by default, while Terraform stores them in plaintext unless you use external secret management.
Terraform has a massive module registry and broader community adoption, especially among operations-focused teams. Pulumi has a growing ecosystem and can actually consume any Terraform provider via its Pulumi-Terraform bridge, so provider coverage is effectively equivalent. However, reusable module availability is still stronger in Terraform.
Pulumi's biggest advantage is testability. You can write unit tests using standard test frameworks (Jest, pytest, Go testing) that mock cloud resources and verify infrastructure configuration before deployment. Terraform testing with terraform test or Terratest is possible but less ergonomic. For organizations with strong software engineering culture, Pulumi enables treating infrastructure truly as software.
Choose Pulumi when your team has strong TypeScript/Python skills, you need complex logic (dynamic resource generation, conditional deployments), or you want deep integration with application code. Choose Terraform when you need the broadest ecosystem, your team prefers declarative simplicity, or you're in a regulated environment where HCL's constraints are a feature.
Code Example
# Pulumi TypeScript - Dynamic resource creation with real language features
import * as pulumi from "@pulumi/pulumi";
import * as aws from "@pulumi/aws";
interface ServiceConfig {
name: string;
port: number;
replicas: number;
isPublic: boolean;
}
const services: ServiceConfig[] = [
{ name: "api", port: 8080, replicas: 3, isPublic: true },
{ name: "worker", port: 9090, replicas: 5, isPublic: false },
{ name: "auth", port: 8443, replicas: 2, isPublic: true },
];
// Type-safe loop with conditionals - natural in TypeScript
for (const svc of services) {
const sg = new aws.ec2.SecurityGroup(`${svc.name}-sg`, {
ingress: [{
protocol: "tcp",
fromPort: svc.port,
toPort: svc.port,
cidrBlocks: svc.isPublic ? ["0.0.0.0/0"] : ["10.0.0.0/8"],
}],
});
const asg = new aws.autoscaling.Group(`${svc.name}-asg`, {
desiredCapacity: svc.replicas,
minSize: Math.ceil(svc.replicas / 2), // Real math, not HCL hacks
maxSize: svc.replicas * 2,
tags: [{ key: "Service", value: svc.name, propagateAtLaunch: true }],
});
}
# Equivalent Terraform (for comparison) requires more boilerplate:
# variable "services" { type = map(object({...})) }
# resource "aws_security_group" "svc" {
# for_each = var.services
# ingress { cidr_blocks = each.value.is_public ? ["0.0.0.0/0"] : ["10.0.0.0/8"] }
# }Interview Tip
Don't bash Terraform. Acknowledge its strengths (ecosystem, declarative safety) while articulating specific scenarios where Pulumi's programming language approach solves real problems: complex conditional logic, type safety, and testability.
💬 Comments
Quick Answer
Use Pulumi Cloud or a self-hosted S3 backend with per-stack state files, organize stacks as <project>/<environment>, implement state locking, and maintain backup/export procedures for corruption recovery using `pulumi stack export/import`.
Detailed Answer
Pulumi supports multiple state backends: Pulumi Cloud (SaaS), self-hosted Pulumi Cloud (Enterprise), S3-compatible storage, Azure Blob, Google Cloud Storage, or local filesystem. For enterprise use, Pulumi Cloud provides built-in encryption, audit logging, RBAC, state history, and concurrent operation locking. If you must self-host, use S3 with versioning enabled, server-side encryption (SSE-KMS), and DynamoDB for state locking (configured via pulumi login s3://bucket-name).
Organize stacks using the pattern <org>/<project>/<stack>. For 50+ microservices, create a project per service with stacks per environment: acme/order-service/dev, acme/order-service/prod. Shared infrastructure (VPC, DNS, databases) gets its own project with stack references for cross-stack outputs. This isolation ensures a bad deployment in one service cannot corrupt another service's state.
State corruption typically happens when a deployment is interrupted (SIGKILL, network failure) or when someone manually modifies cloud resources outside Pulumi. Recovery workflow: First, export the corrupted state with pulumi stack export > state.json. Examine the JSON to identify the corruption (orphaned resources, incorrect outputs, duplicate URNs). Edit the JSON to fix issues, then import with pulumi stack import < state.json. For resources that exist in the cloud but not in state, use pulumi import to bring them under management. For resources in state but deleted from cloud, remove them from the exported JSON.
Enable S3 bucket versioning so every state change is recoverable. Run pulumi stack export as a pre-deployment step in CI/CD and archive state snapshots. Use Pulumi Cloud's built-in state history to compare state before and after deployments. Implement stack policies that prevent deletion of critical stacks without explicit confirmation.
Code Example
# Configure self-hosted S3 backend
pulumi login s3://pulumi-state-prod?region=us-east-1
# Stack organization for microservices
pulumi stack init acme/order-service/dev
pulumi stack init acme/order-service/staging
pulumi stack init acme/order-service/prod
# Export state for backup (run in CI before every deploy)
pulumi stack export --stack acme/order-service/prod > \
backups/order-service-prod-$(date +%Y%m%d-%H%M%S).json
# State corruption recovery
pulumi stack export --stack acme/order-service/prod > state.json
# Examine state - find orphaned resources
jq '.deployment.resources[] | select(.type == "aws:ec2/instance:Instance") | .urn' state.json
# Remove a resource that was manually deleted from AWS
jq 'del(.deployment.resources[] | select(.urn | contains("orphaned-instance")))' \
state.json > state-fixed.json
pulumi stack import --stack acme/order-service/prod < state-fixed.json
# Import an existing AWS resource into Pulumi state
pulumi import aws:s3/bucket:Bucket my-bucket my-existing-bucket-name
# Pulumi state move between stacks (restructuring)
pulumi state move --source acme/monolith/prod --dest acme/order-service/prod \
'urn:pulumi:prod::monolith::aws:s3/bucket:Bucket::order-bucket'
# CI/CD state backup with retention
#!/bin/bash
STATE_BUCKET="s3://pulumi-state-backups"
STACK="acme/order-service/prod"
pulumi stack export --stack $STACK | \
aws s3 cp - "${STATE_BUCKET}/${STACK}/$(date +%Y%m%d-%H%M%S).json"
# Clean backups older than 90 days
aws s3 ls "${STATE_BUCKET}/${STACK}/" | \
awk '{print $4}' | head -n -90 | \
xargs -I{} aws s3 rm "${STATE_BUCKET}/${STACK}/{}"Interview Tip
Emphasize that state corruption is an operational reality, not a hypothetical. Show you have a recovery playbook: export, diagnose, fix, import. Mention S3 versioning as a safety net and automated pre-deployment state backups in CI/CD.
💬 Comments
Quick Answer
Use Pulumi StackReference to export outputs from the platform stack (VPC ID, subnet IDs, cluster endpoint) and consume them in application stacks. Enforce contracts via typed interfaces, version outputs, and use Pulumi ESC for environment-level configuration sharing.
Detailed Answer
Structure your Pulumi projects in layers: Layer 0 (networking/VPC), Layer 1 (compute/EKS, databases/RDS), Layer 2 (platform services like ingress, monitoring), Layer 3 (application workloads). Each layer is a separate Pulumi project with its own state, owned by the appropriate team. Upper layers reference lower layers via StackReference, creating a dependency graph without tight coupling.
The platform team exports outputs from their stack using pulumi.export(). Application teams consume these using new pulumi.StackReference('org/infra/prod') and call getOutput('vpcId'). Critically, stack references return Output<T> values that are resolved asynchronously during deployment, maintaining Pulumi's dependency tracking. If the referenced stack's output changes, dependent stacks see the new value on their next deployment.
The danger with cross-stack references is breaking changes: the platform team renames an output and 20 application stacks break. Mitigate this by defining a typed interface for stack outputs, versioning the contract, and running integration tests that verify outputs exist before merging platform changes. In TypeScript, create a shared npm package defining the output interface that both platform and application code import.
Stack references create implicit deployment ordering. The platform stack must be deployed before application stacks. Implement this in CI/CD with explicit pipeline stages. Use pulumi preview in application stacks after platform changes to detect breakage before applying. Consider using Pulumi ESC (Environments, Secrets, Configuration) as a higher-level abstraction for sharing environment configuration that doesn't require direct stack-to-stack references.
Code Example
// === Platform Team: infra/index.ts ===
import * as pulumi from "@pulumi/pulumi";
import * as aws from "@pulumi/aws";
const vpc = new aws.ec2.Vpc("platform-vpc", {
cidrBlock: "10.0.0.0/16",
enableDnsHostnames: true,
});
const cluster = new aws.eks.Cluster("platform-eks", {
vpcConfig: { subnetIds: privateSubnetIds },
});
// Export outputs - these form the contract
pulumi.export("vpcId", vpc.id);
pulumi.export("privateSubnetIds", privateSubnetIds);
pulumi.export("eksClusterName", cluster.name);
pulumi.export("eksClusterEndpoint", cluster.endpoint);
pulumi.export("eksOidcProviderArn", cluster.identities[0].oidcs[0].issuer);
// === Application Team: order-service/index.ts ===
import * as pulumi from "@pulumi/pulumi";
import * as aws from "@pulumi/aws";
import * as k8s from "@pulumi/kubernetes";
const env = pulumi.getStack(); // "dev", "prod"
const infra = new pulumi.StackReference(`acme/infra/${env}`);
// Consume platform outputs with type safety
const vpcId = infra.getOutput("vpcId") as pulumi.Output<string>;
const subnetIds = infra.getOutput("privateSubnetIds") as pulumi.Output<string[]>;
const clusterName = infra.getOutput("eksClusterName") as pulumi.Output<string>;
// Use platform resources
const appSg = new aws.ec2.SecurityGroup("order-service-sg", {
vpcId: vpcId,
ingress: [{ protocol: "tcp", fromPort: 8080, toPort: 8080, cidrBlocks: ["10.0.0.0/16"] }],
});
// === Shared contract package: @acme/infra-contracts ===
export interface PlatformOutputs {
vpcId: string;
privateSubnetIds: string[];
eksClusterName: string;
eksClusterEndpoint: string;
eksOidcProviderArn: string;
}
// Type-safe helper
export function getPlatformOutputs(stack: pulumi.StackReference): {
[K in keyof PlatformOutputs]: pulumi.Output<PlatformOutputs[K]>
} {
return {
vpcId: stack.getOutput("vpcId") as pulumi.Output<string>,
privateSubnetIds: stack.getOutput("privateSubnetIds") as pulumi.Output<string[]>,
eksClusterName: stack.getOutput("eksClusterName") as pulumi.Output<string>,
eksClusterEndpoint: stack.getOutput("eksClusterEndpoint") as pulumi.Output<string>,
eksOidcProviderArn: stack.getOutput("eksOidcProviderArn") as pulumi.Output<string>,
};
}Interview Tip
Emphasize the contract pattern: define a typed interface for cross-stack outputs and treat output changes as API breaking changes. Mention deployment ordering in CI/CD as an operational concern that many candidates overlook.
💬 Comments
Quick Answer
The Pulumi Automation API embeds Pulumi's deployment engine as a library, allowing you to create, configure, and deploy stacks programmatically from application code. Build a REST API that accepts infrastructure requests, runs `stack.up()` asynchronously, and returns status via webhooks or polling.
Detailed Answer
The Pulumi Automation API is a programmatic interface to Pulumi's engine, available in TypeScript, Python, Go, and C#. Instead of running pulumi up from the CLI, you call LocalWorkspace.createOrSelectStack() and stack.up() from your application code. This enables building platforms, CI/CD integrations, and self-service portals that provision infrastructure without shelling out to CLI commands.
Build a REST API (Express/FastAPI/Go) that accepts infrastructure requests (e.g., 'create a dev environment with RDS and Redis'). The API validates the request, creates a Pulumi stack with the appropriate configuration, runs stack.up() asynchronously in a background worker, and stores the deployment status in a database. Users poll for status or receive a webhook callback when provisioning completes. Each user's environment gets its own Pulumi stack for isolation.
Inline programs let you define infrastructure as a function passed directly to the Automation API, rather than requiring a separate Pulumi project directory. Stack configuration is set programmatically via stack.setConfig(). You can stream deployment logs in real-time by passing a callback to stack.up({ onOutput: callback }). The stack.outputs() method retrieves outputs after deployment. stack.destroy() and stack.workspace.removeStack() handle cleanup.
Run deployments in isolated workers (Kubernetes Jobs or Lambda) to prevent one deployment from affecting others. Implement request queuing (SQS/Redis) to handle burst traffic. Set deployment timeouts. Store stack state in a shared backend (Pulumi Cloud or S3) accessible from any worker. Implement RBAC to control who can provision what. Add cost estimation by running stack.preview() before stack.up() and parsing the resource count.
Code Example
// Self-service infrastructure API using Pulumi Automation API
import * as express from "express";
import { LocalWorkspace, InlineProgramArgs } from "@pulumi/pulumi/automation";
import * as aws from "@pulumi/aws";
const app = express();
// Inline program - defines infrastructure as a function
const createDevEnvironment = async () => {
const vpc = new aws.ec2.Vpc("dev-vpc", { cidrBlock: "10.100.0.0/16" });
const db = new aws.rds.Instance("dev-db", {
engine: "postgres",
instanceClass: "db.t3.micro",
allocatedStorage: 20,
masterUsername: "admin",
masterPassword: new aws.secretsmanager.GetSecretVersionResult({}).secretString,
vpcSecurityGroupIds: [sg.id],
skipFinalSnapshot: true,
});
const redis = new aws.elasticache.Cluster("dev-redis", {
engine: "redis",
nodeType: "cache.t3.micro",
numCacheNodes: 1,
});
return { dbEndpoint: db.endpoint, redisEndpoint: redis.cacheNodes[0].address };
};
app.post("/api/environments", async (req, res) => {
const { userId, envName, tier } = req.body;
const stackName = `dev-${userId}-${envName}`;
// Create stack programmatically
const stack = await LocalWorkspace.createOrSelectStack({
stackName,
projectName: "dev-environments",
program: createDevEnvironment,
});
// Set stack configuration
await stack.setConfig("aws:region", { value: "us-west-2" });
await stack.setConfig("tier", { value: tier });
// Deploy asynchronously
res.json({ stackName, status: "provisioning" });
try {
const result = await stack.up({
onOutput: (msg) => console.log(`[${stackName}] ${msg}`),
});
// Store result in database
await db.saveDeployment(stackName, {
status: "ready",
outputs: result.outputs,
duration: result.summary.duration,
});
} catch (err) {
await db.saveDeployment(stackName, { status: "failed", error: err.message });
}
});
// Destroy environment on demand
app.delete("/api/environments/:stackName", async (req, res) => {
const stack = await LocalWorkspace.selectStack({
stackName: req.params.stackName,
projectName: "dev-environments",
program: createDevEnvironment,
});
await stack.destroy({ onOutput: console.log });
await stack.workspace.removeStack(req.params.stackName);
res.json({ status: "destroyed" });
});Interview Tip
The Automation API is Pulumi's killer feature that Terraform lacks. Emphasize that it turns infrastructure provisioning into an API call, enabling self-service platforms. Discuss worker isolation and async processing for production readiness.
💬 Comments
Quick Answer
Use Pulumi's mocking framework for unit tests to verify resource configurations without cloud calls, property tests to validate outputs, and integration tests that deploy real stacks to ephemeral environments. Layer these in CI/CD with unit tests on every PR and integration tests on merge to main.
Detailed Answer
Pulumi provides a mocking framework that intercepts resource creation calls and returns fake outputs. You define mock implementations that simulate cloud provider behavior without making any API calls. This lets you test resource configuration logic, conditional resource creation, and output transformations in milliseconds. In TypeScript, use pulumi.runtime.setMocks() in your test setup. In Python, use pulumi.runtime.set_mocks().
Property tests verify that the outputs of your Pulumi program meet certain invariants after the mock engine processes all resources. For example: all S3 buckets must have encryption enabled, all security groups must not allow 0.0.0.0/0 ingress on port 22, all RDS instances must have multi-AZ enabled in production. These are like unit tests but focus on compliance properties rather than specific resource configurations.
Integration tests deploy real infrastructure to an ephemeral environment (e.g., a dedicated AWS account or namespace), verify the resources work correctly, and tear everything down. Pulumi's @pulumi/pulumi/automation API makes this programmatic: create a temporary stack, deploy it, run assertions against real cloud resources (check if the load balancer responds, verify database connectivity), then destroy the stack. These tests are slow (minutes) but catch provider-specific issues that mocks miss.
Structure your pipeline in layers: unit tests and property tests run on every pull request (seconds). Integration tests run on merge to main against a staging environment (minutes). Canary deployments to production run after integration tests pass. Use pulumi preview as an additional check in PRs to detect potential issues without deploying.
Code Example
// === Unit Test with Mocks (Jest + TypeScript) ===
import * as pulumi from "@pulumi/pulumi";
// Set up mocks BEFORE importing the Pulumi program
pulumi.runtime.setMocks({
newResource: function(args: pulumi.runtime.MockResourceArgs): { id: string, state: any } {
return {
id: args.inputs.name + "-id",
state: {
...args.inputs,
arn: `arn:aws:s3:::${args.inputs.name || "test"}`,
},
};
},
call: function(args: pulumi.runtime.MockCallArgs) {
return args.inputs;
},
});
// Now import the infrastructure module
import { bucket, securityGroup } from "../index";
describe("Infrastructure", () => {
test("S3 bucket has encryption enabled", async () => {
const encryption = await new Promise<any>((resolve) =>
bucket.serverSideEncryptionConfiguration.apply(resolve)
);
expect(encryption).toBeDefined();
expect(encryption.rules[0].applyServerSideEncryptionByDefault.sseAlgorithm)
.toBe("aws:kms");
});
test("Security group does not allow SSH from anywhere", async () => {
const ingress = await new Promise<any[]>((resolve) =>
securityGroup.ingress.apply(resolve)
);
const sshRules = ingress.filter(r => r.fromPort === 22);
for (const rule of sshRules) {
expect(rule.cidrBlocks).not.toContain("0.0.0.0/0");
}
});
});
// === Policy/Property Test ===
import { PolicyPack, validateResourceOfType } from "@pulumi/policy";
import * as aws from "@pulumi/aws";
new PolicyPack("compliance", {
policies: [{
name: "s3-no-public-read",
description: "S3 buckets must not have public-read ACL",
enforcementLevel: "mandatory",
validateResource: validateResourceOfType(aws.s3.Bucket, (bucket, args, report) => {
if (bucket.acl === "public-read") {
report("S3 bucket must not be public-read");
}
}),
}],
});
// === Integration Test (Automation API) ===
import { LocalWorkspace } from "@pulumi/pulumi/automation";
import axios from "axios";
describe("Integration", () => {
let stack: any;
beforeAll(async () => {
stack = await LocalWorkspace.createOrSelectStack({
stackName: `test-${Date.now()}`,
workDir: "../",
});
await stack.setConfig("aws:region", { value: "us-west-2" });
await stack.up();
}, 300000);
test("ALB responds with 200", async () => {
const outputs = await stack.outputs();
const resp = await axios.get(outputs.albDnsName.value);
expect(resp.status).toBe(200);
});
afterAll(async () => {
await stack.destroy();
await stack.workspace.removeStack(stack.name);
}, 300000);
});Interview Tip
Testing is where Pulumi genuinely outshines Terraform. Show the three layers: fast unit tests with mocks, property/policy tests for compliance, and slow integration tests with real resources. Mention the CI/CD pyramid where unit tests gate PRs and integration tests gate merges.
💬 Comments
Quick Answer
Create a ComponentResource subclass that encapsulates all child resources, accepts a typed args interface, registers child resources with the parent, and exposes outputs. Publish it as an npm/PyPI package for team consumption with versioning and documentation.
Detailed Answer
Pulumi's ComponentResource is the abstraction mechanism for creating reusable, higher-level resources. Unlike a regular resource that maps to a single cloud API, a ComponentResource groups multiple resources into a logical unit with a single URN. Child resources are registered with { parent: this } so they appear nested in the Pulumi state tree. The component defines its own typed input interface and exposes outputs.
Follow the principle of sensible defaults with escape hatches. The component should work out of the box for 80% of use cases with minimal configuration (just service name, container image, port), while allowing power users to override every aspect (custom health check paths, CPU/memory, scaling policies). Use TypeScript interfaces with optional fields and spread operators to merge defaults with user overrides.
ComponentResources don't have their own cloud-side state; they're purely logical groupings. However, they do appear in Pulumi state with their own URN. When you update a component (add a new child resource), Pulumi handles the diff correctly because each child resource has its own URN. Deleting a component deletes all child resources. This is critical for cleanup in self-service scenarios.
Publish your component as an npm package (TypeScript) or PyPI package (Python) with semantic versioning. Breaking changes to the component's interface get a major version bump. Include README documentation with usage examples. Use Pulumi's schema system if you want to create a multi-language component that works across TypeScript, Python, Go, and C# simultaneously.
Code Example
import * as pulumi from "@pulumi/pulumi";
import * as aws from "@pulumi/aws";
export interface MicroserviceArgs {
image: pulumi.Input<string>;
port: number;
cpu?: number;
memory?: number;
desiredCount?: number;
healthCheckPath?: string;
environment?: Record<string, pulumi.Input<string>>;
vpcId: pulumi.Input<string>;
subnetIds: pulumi.Input<string>[];
albListenerArn: pulumi.Input<string>;
hostHeader: string;
hostedZoneId: pulumi.Input<string>;
}
export class Microservice extends pulumi.ComponentResource {
public readonly url: pulumi.Output<string>;
public readonly serviceArn: pulumi.Output<string>;
public readonly targetGroupArn: pulumi.Output<string>;
constructor(name: string, args: MicroserviceArgs, opts?: pulumi.ComponentResourceOptions) {
super("acme:platform:Microservice", name, {}, opts);
const cpu = args.cpu || 256;
const memory = args.memory || 512;
const desiredCount = args.desiredCount || 2;
const healthCheckPath = args.healthCheckPath || "/health";
// Task Definition
const taskDef = new aws.ecs.TaskDefinition(`${name}-task`, {
family: name,
networkMode: "awsvpc",
requiresCompatibilities: ["FARGATE"],
cpu: cpu.toString(),
memory: memory.toString(),
executionRoleArn: executionRole.arn,
containerDefinitions: pulumi.output(args.image).apply(image => JSON.stringify([{
name: name,
image: image,
portMappings: [{ containerPort: args.port }],
environment: Object.entries(args.environment || {}).map(([k, v]) => ({ name: k, value: v })),
logConfiguration: {
logDriver: "awslogs",
options: { "awslogs-group": `/ecs/${name}`, "awslogs-region": "us-east-1" },
},
}])),
}, { parent: this });
// Target Group
const tg = new aws.lb.TargetGroup(`${name}-tg`, {
port: args.port,
protocol: "HTTP",
vpcId: args.vpcId,
targetType: "ip",
healthCheck: { path: healthCheckPath, interval: 15 },
}, { parent: this });
// ALB Listener Rule
new aws.lb.ListenerRule(`${name}-rule`, {
listenerArn: args.albListenerArn,
conditions: [{ hostHeader: { values: [args.hostHeader] } }],
actions: [{ type: "forward", targetGroupArn: tg.arn }],
}, { parent: this });
// ECS Service
const service = new aws.ecs.Service(`${name}-svc`, {
cluster: clusterArn,
taskDefinition: taskDef.arn,
desiredCount: desiredCount,
launchType: "FARGATE",
networkConfiguration: {
subnets: args.subnetIds,
assignPublicIp: false,
},
loadBalancers: [{ targetGroupArn: tg.arn, containerName: name, containerPort: args.port }],
}, { parent: this });
// CloudWatch Alarm
new aws.cloudwatch.MetricAlarm(`${name}-5xx-alarm`, {
metricName: "HTTPCode_Target_5XX_Count",
namespace: "AWS/ApplicationELB",
statistic: "Sum",
period: 60,
evaluationPeriods: 3,
threshold: 10,
comparisonOperator: "GreaterThanThreshold",
alarmActions: [snsTopicArn],
}, { parent: this });
this.url = pulumi.interpolate`https://${args.hostHeader}`;
this.serviceArn = service.id;
this.targetGroupArn = tg.arn;
this.registerOutputs({ url: this.url, serviceArn: this.serviceArn });
}
}
// Usage:
const orderService = new Microservice("order-service", {
image: "123456789.dkr.ecr.us-east-1.amazonaws.com/order-service:v1.2.3",
port: 8080,
desiredCount: 3,
hostHeader: "orders.api.acme.com",
vpcId: infra.vpcId,
subnetIds: infra.privateSubnetIds,
albListenerArn: infra.albListenerArn,
hostedZoneId: infra.hostedZoneId,
});Interview Tip
Show that ComponentResource is Pulumi's answer to Terraform modules but with real OOP principles: inheritance, typed interfaces, encapsulation. Emphasize the { parent: this } pattern for state tree organization and the registerOutputs call for proper resource tracking.
💬 Comments
Quick Answer
CrossGuard is Pulumi's policy-as-code framework that runs PolicyPacks (written in TypeScript or Python) during `pulumi preview` and `pulumi up`. Policies validate resource configurations before deployment, with enforcement levels of advisory, mandatory, or disabled. Unlike OPA, CrossGuard has native access to Pulumi resource types and outputs.
Detailed Answer
CrossGuard PolicyPacks are collections of rules that run against the resource graph during Pulumi operations. Each policy can inspect individual resources (validateResource), the entire stack (validateStack), or resource configurations before creation. Policies run locally during pulumi up or can be enforced server-side via Pulumi Cloud, where the organization admin configures which PolicyPacks are mandatory for which stacks.
Three levels control what happens when a policy violation is detected: advisory logs a warning but allows deployment to proceed, mandatory blocks deployment entirely, and disabled skips the policy. In practice, start new policies as advisory to measure impact, then promote to mandatory after teams have remediated violations. Pulumi Cloud allows setting enforcement levels per stack, so you might enforce strict policies in production but keep them advisory in development.
OPA (Open Policy Agent) uses Rego, a purpose-built policy language. It's provider-agnostic and can evaluate Terraform plans, Kubernetes manifests, and API requests. Sentinel is HashiCorp's policy framework, tightly integrated with Terraform Cloud/Enterprise. CrossGuard's advantage is that policies are written in the same language as your infrastructure code (TypeScript/Python), leveraging IDE support and type checking. Its disadvantage is that it only works with Pulumi. For organizations standardized on Pulumi, CrossGuard is the natural choice. For multi-tool environments, OPA provides broader coverage.
Stack-level policies can enforce architectural constraints like 'every stack must have at least one CloudWatch alarm' or 'total estimated monthly cost must not exceed $5000'. You can combine CrossGuard with Pulumi ESC to apply different policy sets based on environment (strict for prod, relaxed for dev). Custom remediation advisories can suggest the exact fix for violations.
Code Example
// PolicyPack: compliance-policies/index.ts
import { PolicyPack, validateResourceOfType, StackValidationPolicy } from "@pulumi/policy";
import * as aws from "@pulumi/aws";
new PolicyPack("acme-compliance", {
policies: [
// Resource-level: No public S3 buckets
{
name: "s3-no-public-access",
description: "S3 buckets must block all public access",
enforcementLevel: "mandatory",
validateResource: validateResourceOfType(aws.s3.BucketPublicAccessBlock, (block, args, report) => {
if (!block.blockPublicAcls || !block.blockPublicPolicy ||
!block.ignorePublicAcls || !block.restrictPublicBuckets) {
report("All public access block settings must be enabled");
}
}),
},
// Resource-level: RDS must be encrypted and multi-AZ in prod
{
name: "rds-encryption-required",
description: "RDS instances must use encryption at rest",
enforcementLevel: "mandatory",
validateResource: validateResourceOfType(aws.rds.Instance, (db, args, report) => {
if (!db.storageEncrypted) {
report("RDS instance must have storageEncrypted = true");
}
if (args.getStack().includes("prod") && !db.multiAz) {
report("Production RDS must be multi-AZ");
}
}),
},
// Resource-level: EC2 instances must use approved AMIs
{
name: "ec2-approved-amis",
description: "EC2 instances must use company-approved AMIs",
enforcementLevel: "mandatory",
validateResource: validateResourceOfType(aws.ec2.Instance, (instance, args, report) => {
const approvedAmis = ["ami-0abc123", "ami-0def456", "ami-0ghi789"];
if (instance.ami && !approvedAmis.includes(instance.ami)) {
report(`AMI ${instance.ami} is not approved. Use one of: ${approvedAmis.join(", ")}`);
}
}),
},
// Stack-level: Every stack must have monitoring
{
name: "require-monitoring",
description: "Every production stack must include CloudWatch alarms",
enforcementLevel: "mandatory",
validateStack: (args, report) => {
if (!args.getStack().includes("prod")) return;
const alarms = args.resources.filter(
r => r.type === "aws:cloudwatch/metricAlarm:MetricAlarm"
);
if (alarms.length === 0) {
report("Production stacks must include at least one CloudWatch alarm");
}
},
} as StackValidationPolicy,
],
});
# Run policies locally
pulumi up --policy-pack ./compliance-policies
# Publish to Pulumi Cloud for server-side enforcement
pulumi policy publish acme-compliance
# Enable for all production stacks in the org
pulumi policy enable acme/acme-compliance --enforcement-level mandatoryInterview Tip
Compare CrossGuard with OPA objectively: CrossGuard is Pulumi-native with language advantages, OPA is universal but requires Rego expertise. Mention the advisory-to-mandatory promotion workflow as a practical governance strategy that avoids blocking developers unnecessarily.
💬 Comments
Quick Answer
A dynamic provider implements create, read, update, and delete operations as code functions that Pulumi's engine calls during deployments. Define a class extending `pulumi.dynamic.ResourceProvider` with CRUD methods, then create a `pulumi.dynamic.Resource` subclass that uses it.
Detailed Answer
Pulumi's built-in providers (AWS, Azure, GCP, Kubernetes) cover major cloud platforms, but organizations often need to manage resources in custom systems: internal service registries, legacy CMDB databases, DNS appliances, or third-party SaaS APIs without Pulumi providers. Dynamic providers let you write CRUD logic in your Pulumi program's language (TypeScript/Python) without building a full provider binary.
A dynamic provider is a class implementing the ResourceProvider interface with methods: create(inputs) returns the new resource's ID and output properties, read(id, props) fetches current state, update(id, olds, news) modifies the resource, delete(id, props) removes it, and optionally diff(id, olds, news) determines if an update is needed. Pulumi's engine calls these methods at the appropriate lifecycle points. Each method must be idempotent.
Dynamic providers serialize the provider code into Pulumi state, so they cannot reference external dependencies or closures that aren't serializable. For complex providers, consider building a native Pulumi provider using the Pulumi Provider SDK (Go/Python), which generates a proper provider binary with schema, documentation, and multi-language support. Dynamic providers are best for internal tooling, prototypes, or simple integrations.
In production, dynamic providers are commonly used for: registering services in Consul/Eureka, creating DNS records in legacy BIND servers via API, provisioning accounts in internal identity systems, or managing configuration in proprietary management platforms. Always implement proper error handling and idempotency in CRUD methods.
Code Example
import * as pulumi from "@pulumi/pulumi";
import axios from "axios";
// Dynamic provider for an internal service registry
interface ServiceRegistryInputs {
serviceName: string;
serviceUrl: string;
healthCheckPath: string;
team: string;
environment: string;
}
const serviceRegistryProvider: pulumi.dynamic.ResourceProvider = {
async create(inputs: ServiceRegistryInputs): Promise<pulumi.dynamic.CreateResult> {
// Call internal API to register service
const response = await axios.post("https://registry.internal.acme.com/api/v1/services", {
name: inputs.serviceName,
url: inputs.serviceUrl,
health_check: inputs.healthCheckPath,
team: inputs.team,
environment: inputs.environment,
}, {
headers: { "Authorization": `Bearer ${process.env.REGISTRY_TOKEN}` },
});
return {
id: response.data.id,
outs: {
...inputs,
registryId: response.data.id,
registeredAt: response.data.created_at,
},
};
},
async read(id: string, props: any): Promise<pulumi.dynamic.ReadResult> {
const response = await axios.get(
`https://registry.internal.acme.com/api/v1/services/${id}`,
{ headers: { "Authorization": `Bearer ${process.env.REGISTRY_TOKEN}` } }
);
return {
id: id,
props: {
serviceName: response.data.name,
serviceUrl: response.data.url,
healthCheckPath: response.data.health_check,
team: response.data.team,
environment: response.data.environment,
registryId: id,
registeredAt: response.data.created_at,
},
};
},
async update(id: string, olds: any, news: ServiceRegistryInputs): Promise<pulumi.dynamic.UpdateResult> {
await axios.put(`https://registry.internal.acme.com/api/v1/services/${id}`, {
name: news.serviceName,
url: news.serviceUrl,
health_check: news.healthCheckPath,
team: news.team,
}, {
headers: { "Authorization": `Bearer ${process.env.REGISTRY_TOKEN}` },
});
return { outs: { ...news, registryId: id } };
},
async delete(id: string, props: any): Promise<void> {
await axios.delete(
`https://registry.internal.acme.com/api/v1/services/${id}`,
{ headers: { "Authorization": `Bearer ${process.env.REGISTRY_TOKEN}` } }
);
},
};
// Resource class using the dynamic provider
class ServiceRegistration extends pulumi.dynamic.Resource {
public readonly registryId!: pulumi.Output<string>;
public readonly registeredAt!: pulumi.Output<string>;
constructor(name: string, args: ServiceRegistryInputs, opts?: pulumi.CustomResourceOptions) {
super(serviceRegistryProvider, name, { ...args, registryId: undefined, registeredAt: undefined }, opts);
}
}
// Usage
const registration = new ServiceRegistration("order-service-reg", {
serviceName: "order-service",
serviceUrl: "https://orders.internal.acme.com",
healthCheckPath: "/health",
team: "commerce",
environment: pulumi.getStack(),
});Interview Tip
Dynamic providers show deep Pulumi knowledge. Explain the serialization limitation upfront and when to graduate to a native provider. Emphasize idempotency in CRUD methods - if create is called twice with the same inputs, it should be safe.
💬 Comments
Quick Answer
Pulumi ESC provides hierarchical environment definitions that compose and inherit from each other. Define base configurations, layer environment-specific overrides, integrate with external secret stores (Vault, AWS Secrets Manager), and consume them in Pulumi stacks, CLI tools, and application code through `esc run`.
Detailed Answer
Pulumi ESC (Environments, Secrets, Configuration) is a centralized configuration management service separate from Pulumi IaC. It defines 'environments' as YAML documents containing key-value pairs, secrets, and references to external secret stores. Environments can import from other environments, creating a composition hierarchy. Unlike traditional .env files, ESC provides versioning, RBAC, audit logging, and integration with multiple secret backends.
Design your environments in layers: a base environment with defaults shared across all services, a base/aws-<region> with region-specific AWS credentials, a <service>/base with service-specific config, and <service>/<env> for environment-specific overrides. The composition order determines precedence - later imports override earlier ones. This reduces duplication dramatically: database connection strings, API URLs, and feature flags are defined once in the appropriate layer.
ESC integrates natively with AWS Secrets Manager, HashiCorp Vault, Azure Key Vault, GCP Secret Manager, and 1Password. Instead of copying secrets into ESC, you define references: fn::secret::aws-secrets-manager: { secretId: 'prod/db-password' }. ESC resolves these at runtime, and secrets never leave the source store. This satisfies compliance requirements that mandate secret stores as the source of truth.
ESC values can be consumed in three ways: (1) in Pulumi IaC stacks via pulumi config env add <env-name> which injects ESC values as stack config, (2) from the CLI via esc run <env> -- <command> which sets environment variables for any process, and (3) via the ESC SDK/API for programmatic access. This means ESC is not limited to Pulumi deployments - it can configure Terraform runs, Docker containers, CI/CD pipelines, and local development environments.
Code Example
# === Environment Hierarchy ===
# base.yaml - shared across all services and environments
# pulumi esc env init acme/base
values:
app:
region: us-east-1
log_level: info
metrics_endpoint: https://datadog.internal.acme.com
aws:
fn::open::aws-login:
oidc:
roleArn: arn:aws:iam::123456789:role/pulumi-esc
sessionName: pulumi-esc
# base-prod.yaml - production-specific overrides
# pulumi esc env init acme/base-prod
imports:
- acme/base
values:
app:
log_level: warn
replicas: 3
multi_az: true
pulumiConfig:
aws:region: us-east-1
# order-service/prod.yaml - service + env specific
# pulumi esc env init acme/order-service-prod
imports:
- acme/base-prod
values:
app:
replicas: 5
feature_flags:
new_checkout: true
v2_api: false
secrets:
db_password:
fn::secret:
fn::open::aws-secrets:
get:
secretId: prod/order-service/db-password
api_key:
fn::secret:
fn::open::vault-secrets:
address: https://vault.internal.acme.com
path: secret/data/order-service/api-key
field: value
environmentVariables:
DATABASE_URL: postgres://orders:${secrets.db_password}@db.prod.acme.com:5432/orders
API_KEY: ${secrets.api_key}
LOG_LEVEL: ${app.log_level}
REPLICAS: ${app.replicas}
# === Consume in Pulumi stack ===
# Pulumi.prod.yaml
config:
pulumi:esc: acme/order-service-prod
# === Consume via CLI for any tool ===
# Run a command with ESC environment variables injected
esc run acme/order-service-prod -- docker compose up
esc run acme/order-service-prod -- terraform apply
esc run acme/order-service-prod -- kubectl apply -f deploy.yaml
# Open an interactive shell with ESC variables
esc open acme/order-service-prod --format shell
# List all environments
esc env ls acme
# Compare environments
esc env diff acme/order-service-dev acme/order-service-prodInterview Tip
ESC is newer but strategically important in the Pulumi ecosystem. Emphasize the composition pattern (imports hierarchy) that eliminates config duplication, and the fact that ESC is not Pulumi-only - it can inject env vars into any tool. The external secret store integration (fn::open) is the key enterprise feature.
💬 Comments
Quick Answer
Optimize by splitting large stacks into smaller ones by service boundary, using `--target` for targeted updates, enabling parallel resource operations, caching provider plugins, and implementing preview-on-PR with apply-on-merge workflows with proper state locking and drift detection.
Detailed Answer
A single stack with 200+ resources is a red flag. Split it into logical stacks by service boundary or infrastructure layer: networking (VPC, subnets), compute (EKS, ECS), data (RDS, ElastiCache), and per-service stacks. Each stack deploys independently and faster because Pulumi only refreshes and diffs resources within that stack. Use stack references for cross-stack dependencies. This also reduces blast radius: a bad change to one service's stack doesn't risk the entire infrastructure.
Pulumi deploys resources in parallel by default, respecting dependency order. You can increase parallelism with pulumi up --parallel <N> (default is 10). For independent resources, this dramatically reduces deployment time. Cache provider plugins in CI by persisting ~/.pulumi/plugins between runs. Use pulumi up --refresh=false when you're confident no out-of-band changes occurred (saves the refresh phase). For targeted updates during incident response, use pulumi up --target <urn> to update only specific resources.
Implement a two-phase workflow: pulumi preview runs on every PR and posts the diff as a PR comment for review. pulumi up runs only on merge to main (or on manual approval for production). Use Pulumi's GitHub Actions integration which provides the preview comment natively. Implement environment promotion: dev deploys automatically on merge, staging requires manual approval, production requires two approvals and a change window.
Run pulumi up --diff to see a detailed diff before applying. Enable drift detection by running pulumi refresh periodically (not on every deploy for performance) to detect out-of-band changes. Use --expect-no-changes in scheduled drift detection runs to alert when actual state diverges from desired state. Implement stack policies that prevent accidental deletion of critical resources using the protect resource option.
Code Example
# .github/workflows/pulumi-deploy.yml
name: Pulumi Deploy
on:
pull_request:
branches: [main]
push:
branches: [main]
jobs:
preview:
if: github.event_name == 'pull_request'
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-node@v4
with:
node-version: 20
cache: npm
# Cache Pulumi plugins for faster startup
- uses: actions/cache@v4
with:
path: ~/.pulumi/plugins
key: pulumi-plugins-${{ hashFiles('package-lock.json') }}
- run: npm ci
- uses: pulumi/actions@v5
with:
command: preview
stack-name: acme/order-service/dev
comment-on-pr: true
diff: true
env:
PULUMI_ACCESS_TOKEN: ${{ secrets.PULUMI_ACCESS_TOKEN }}
AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
deploy-dev:
if: github.event_name == 'push' && github.ref == 'refs/heads/main'
runs-on: ubuntu-latest
environment: dev
steps:
- uses: actions/checkout@v4
- uses: actions/setup-node@v4
with:
node-version: 20
cache: npm
- uses: actions/cache@v4
with:
path: ~/.pulumi/plugins
key: pulumi-plugins-${{ hashFiles('package-lock.json') }}
- run: npm ci
- uses: pulumi/actions@v5
with:
command: up
stack-name: acme/order-service/dev
parallel: 20 # Increase parallelism for faster deploys
env:
PULUMI_ACCESS_TOKEN: ${{ secrets.PULUMI_ACCESS_TOKEN }}
deploy-prod:
needs: deploy-dev
runs-on: ubuntu-latest
environment:
name: production
url: https://orders.acme.com
steps:
- uses: pulumi/actions@v5
with:
command: up
stack-name: acme/order-service/prod
parallel: 10 # Conservative parallelism for prod
# Scheduled drift detection
drift-detection:
runs-on: ubuntu-latest
# schedule: cron: '0 6 * * *'
steps:
- uses: pulumi/actions@v5
with:
command: refresh
stack-name: acme/order-service/prod
expect-no-changes: true
# Targeted update for incident response
# pulumi up --target 'urn:pulumi:prod::order-service::aws:ecs/service:Service::order-svc' \
# --parallel 1 --skip-previewInterview Tip
Lead with the stack splitting strategy - it's the highest impact optimization. Then discuss CI/CD workflow design with preview-on-PR and apply-on-merge. Mention drift detection as a production maturity indicator. The --parallel flag and plugin caching are quick wins interviewers appreciate.
💬 Comments
Context
A platform team managed 1,200 cloud resources with Pulumi TypeScript. A provider upgrade introduced schema changes that previewed replacements for load balancers and IAM resources.
Problem
The upgrade looked like a dependency bump, but production previews showed replacement on resources with customer impact. Some resources had custom options that changed replacement order, making the risk harder to see.
Solution
They froze automatic provider upgrades, exported stack state, ran previews per stack, added `protect` to critical resources, and created an upgrade checklist requiring owner approval for any replacement. Safe changes went first; risky resources were handled with explicit migration plans.
Commands
pulumi preview --diff # Reviews provider-driven changes before update
pulumi stack export > prod-stack-backup.json # Captures state before risky migration
pulumi up --target-dependents # Applies scoped changes with dependent awareness
Outcome
The provider upgrade completed over three maintenance windows with no unplanned replacements. Review time increased, but rollback risk dropped sharply.
Lessons Learned
In IaC, dependency upgrades are production changes. The diff is the release note that matters most.
◈ Architecture Diagram
┌──────────┐
│ Learn │
└────┬─────┘
↓
┌──────────┐
│ UseCase │
└────┬─────┘
↓
┌──────────┐
│ Operate │
└──────────┘💬 Comments
Context
A SaaS whose integration tests needed real cloud resources (S3, SQS, RDS) — previously a fixed pool of three shared test environments that teams booked via Slack and regularly left dirty for the next team.
Problem
Shared environments caused test flakiness from leftover state, booking contention serialized teams, and configuration drifted from production because they were hand-patched for two years.
Solution
Built PR-scoped environments with the Pulumi Automation API: a CI job programmatically creates a stack per PR (pr-1234) from the same component library production uses, provisions a namespaced slice (dedicated queues/buckets, shared RDS instance with per-PR database), runs the test suite, and destroys the stack on merge/close. A nightly reaper destroys stacks older than 3 days (force-push orphans). Costs are tagged per-PR so the platform team can see the spend curve.
Commands
stack = auto.create_or_select_stack(f'pr-{pr}', project, program)stack.up(on_output=log); pytest; stack.destroy()
reaper: for s in ws.list_stacks(): if age(s)>3d: destroy
Outcome
Test flakiness from shared state disappeared; environment wait time went from hours (booking) to ~8 minutes (provision); config drift ended because environments are rebuilt from the production component library on every PR. Cost runs ~40% below the old always-on pool.
Lessons Learned
The reaper is not optional — orphaned stacks appeared week one. Sharing the RDS instance (per-PR databases) instead of per-PR instances was the cost/speed compromise that made it viable.
💬 Comments
Context
A platform team whose cloud security posture depended on quarterly audits: findings arrived months after resources were created, and remediation meant chasing teams about infrastructure they'd forgotten deploying.
Problem
Audit-time enforcement is too late — a public bucket exists for a quarter before the report; and audit findings carried no context about why the resource was configured that way, making remediation slow and adversarial.
Solution
Wrote a CrossGuard policy pack enforced org-wide via the Pulumi service: mandatory policies (no public S3 ACLs/website configs, encryption at rest on all storage, no 0.0.0.0/0 ingress except tagged public load balancers, mandatory cost-center/owner tags) run at preview and up, failing the deployment with an explanation and a link to the exception process. Advisory policies (instance-size recommendations, missing backup tags) warn without blocking. Policy violations became PR review comments instead of quarterly findings.
Commands
pulumi policy publish org/security-baseline
pulumi policy enable org/security-baseline latest --policy-group production
policy: validateResourceOfType(aws.s3.Bucket, (b, args, report) => { if (publicAcl(b)) report('S3 buckets must not be public — see EXC-101 for exceptions') })Outcome
Public-bucket findings in the next audit: zero (previous audit: seven). Mean remediation time became zero because violations never deploy; the exception process (annotated, expiring waivers) handled the four legitimate public buckets. Security review shifted from policing to maintaining the policy pack.
Lessons Learned
Advisory-first rollout for each new policy avoided blocking legitimate deploys on day one — two 'obvious' policies had false positives (CDN buckets) that advisory mode surfaced safely. Expiring waivers matter: permanent exceptions become invisible policy holes.
💬 Comments
Symptom
A Pulumi update failed midway and left the database attached to a default parameter group, causing elevated connection errors.
Error Message
error: deleting urn:pulumi:prod::platform::aws:rds/parameterGroup:ParameterGroup::payments-db-pg: ResourceInUse
Root Cause
A resource option forced delete-before-replace on a dependency that could not be safely removed while the database was active. Pulumi was following the declared replacement behavior, but the option violated the service lifecycle. Replacement ordering is a production design decision, not just a diff preference. The underlying issue was a combination of factors that individually seemed harmless but together created a cascading failure. The monitoring that should have caught this early was either missing or configured with thresholds too high to trigger before user impact. The on-call engineer initially pursued the wrong hypothesis because the symptoms resembled a different, more common failure mode. By the time the real root cause was identified, the blast radius had expanded beyond the original service to affect downstream dependencies. This incident exposed a gap in the team's runbooks and highlighted the need for better correlation between metrics, logs, and traces during diagnosis.
Diagnosis Steps
Solution
Remove delete-before-replace for resources that require create-before-delete, restore the expected parameter group, and run a targeted preview before full update. Do not edit state unless provider recovery and import are impossible.
Commands
pulumi preview --diff
pulumi up --target urn:pulumi:prod::platform::aws:rds/parameterGroup:ParameterGroup::payments-db-pg
Prevention
Review replacement options for stateful resources. Use protection on critical infrastructure. Require previews for any update with delete-before-replace.
◈ Architecture Diagram
┌──────────┐
│ Learn │
└────┬─────┘
↓
┌──────────┐
│ Incident │
└────┬─────┘
↓
┌──────────┐
│ Operate │
└──────────┘💬 Comments
Symptom
After a CI runner was OOM-killed mid-`pulumi up`, every following deploy of that stack fails immediately: the state has pending operations, and engineers can't tell whether the interrupted resources (an RDS instance mid-create) actually exist in AWS.
Error Message
error: the current deployment has 1 resource(s) with pending operations: * urn:pulumi:prod::platform::aws:rds/instance:Instance::main-db, interrupted while creating Refusing to proceed. Please run 'pulumi refresh' or 'pulumi stack export | pulumi stack import' after reviewing.
Root Cause
pulumi up was killed between issuing the cloud API call and recording its result, so state says 'creation in flight, outcome unknown'. Pulumi correctly refuses to proceed — continuing could double-create or orphan the resource. The RDS instance had in fact been created (the API call succeeded before the kill), making naive retry a path to a duplicate instance with a name collision at best, a second bill at worst.
Diagnosis Steps
Solution
Ran pulumi refresh, which reconciled state against the cloud: it found the created RDS instance and adopted the completed operation, clearing the pending flag. For a messier historical case (resource half-created and unusable), the fix was stack export, hand-editing the pending operation out after verifying the console, deleting the orphan manually, and re-importing. CI got a bigger runner class for infra jobs plus a trap-based cleanup warning on abnormal exit.
Commands
pulumi refresh --yes --stack prod
aws rds describe-db-instances --db-instance-identifier main-db
pulumi stack export > state.json # inspect pending_operations before surgery
Prevention
Give infra CI jobs generous resources and timeouts (state-mutating jobs should not be OOM candidates), alert on interrupted infra pipelines specifically, and document the refresh/export-import recovery runbook so the first responder doesn't improvise against state. Where supported, use resource names that make accidental duplicates collide loudly rather than silently.
💬 Comments
Symptom
A routine dependency-bump PR (pulumi-aws minor version) shows a green preview in CI summary output ('4 to update'), gets merged, and the overnight deploy attempts to replace the primary API security group — briefly severing traffic while the replacement's rules propagate.
Error Message
Preview (unread detail): aws:ec2/securityGroup:SecurityGroup api-sg [diff: ~revokeRulesOnDelete] requires replacement. Post-incident, the changelog shows the provider changed a default that is immutable-on-update for this resource.
Root Cause
The provider upgrade changed a property default; for security groups that property is replace-only, so Pulumi planned delete-and-recreate. The CI summary ('4 to update') didn't distinguish updates from replacements, the full preview diff wasn't reviewed for a 'just a version bump' PR, and no policy blocked replacement of protected resources. The 2am deploy executed exactly what the plan said.
Diagnosis Steps
Solution
Immediate: pinned the previous provider version and redeployed to restore the original group (the outage was minutes, the cleanup an hour). Durable fixes: protect on crown-jewel resources (protect: true refuses deletion), a CrossGuard policy failing any plan that replaces resources tagged tier:critical, and CI now renders the full preview with replacements called out loudly and requires human acknowledgment of any replace on protected stacks.
Commands
pulumi preview --diff | grep -B2 'requires replacement'
resource opts: {protect: true} # crown jewelspolicy: if (plan.replaces.some(r => r.tags['tier']=='critical')) fail(...)
Prevention
Treat provider upgrades as changes to infrastructure, not dependencies: read the diff, not the summary count. Mark irreplaceable resources with protect and back it with policy (defense in depth). Replacement-aware CI output — a replace is never routine.
💬 Comments