Terraform Architect Interview Questions

Think of a hospital records system. If every department writes to one giant patient file simultaneously, records get corrupted and the wrong medication gets administered. Splitting records by department, locking each file during edits, and storing everything in a central secure archive prevents these disasters. Terraform state management works the same way — it is the record of what infrastructure exists, and mismanaging it causes outages.

Terraform state is a JSON file that maps every resource in your configuration to a real cloud object. When terraform plan runs, it reads the state to determine what exists, compares it to the desired configuration, and calculates the diff. If two engineers run terraform apply simultaneously against the same state, one overwrites the other's changes, causing state corruption where Terraform's view of the world no longer matches reality. Remote backends solve storage and collaboration: S3 stores the state file durably, DynamoDB provides a lock table so only one operation can modify state at a time, and versioning on the S3 bucket enables recovery from bad applies.

Internally, when terraform apply starts, it sends a Lock request to the backend. For S3+DynamoDB, this writes a lock record to the DynamoDB table with a unique ID, the user's identity, and a timestamp. If another process already holds the lock, Terraform exits with an error. After the apply completes, Terraform writes the updated state to S3 and releases the lock. If a process crashes mid-apply, the lock remains until it expires or is manually force-unlocked with terraform force-unlock. Terraform Cloud handles locking internally and adds run queues so multiple plans can exist but only one apply executes at a time per workspace.

At production scale, the critical architectural decision is state splitting. A monolithic state file containing the VPC, databases, Kubernetes clusters, DNS records, and application services means a single terraform apply can accidentally destroy the database while updating a DNS record. The recommended pattern is splitting state by blast radius: network foundations in one state, data layer in another, compute in another, and application configurations in their own states. Each state has its own backend configuration and can use terraform_remote_state data sources or outputs stored in SSM Parameter Store to share values. Workspaces can further separate environments (dev, staging, production) within the same configuration, but they should not be used as a substitute for proper state splitting — all workspaces in a configuration share the same codebase, backend, and permissions.

The non-obvious gotcha is that terraform_remote_state creates a hard coupling between states, and if the upstream state is corrupted or the output names change, downstream plans break. Many mature teams replace terraform_remote_state with data sources that look up infrastructure by tags or names, or they store shared values in AWS SSM Parameter Store or HashiCorp Consul, which decouples state files completely. Another trap is that S3 bucket versioning does not protect against state file deletion — teams must also enable MFA Delete or use S3 Object Lock for regulatory environments.

Think of building a house with prefabricated components. A good prefab wall panel has standard dimensions (clear interface), quality-tested materials (validation), a version number stamped on it (semantic versioning), and is available from a catalog (registry). A bad panel is custom-cut for one house, undocumented, and stored in someone's garage. Terraform module design follows the same principles.

A well-designed Terraform module encapsulates a single infrastructure concern with a clear input/output contract. The module should do one thing well — create an RDS instance with standard security settings, or provision a VPC with consistent CIDR allocation — rather than trying to create an entire environment. Variable validation blocks catch configuration errors at plan time rather than during apply or, worse, at runtime when a database accepts an invalid parameter and fails to start. Validation expressions use conditions and error messages to enforce naming patterns, CIDR ranges, instance size constraints, and environment-specific rules before any API call is made.

Internally, Terraform resolves module sources during terraform init. A module sourced from a Git repository with a version tag (git::https://github.com/company/terraform-aws-rds.git?ref=v2.3.1) is downloaded and cached in .terraform/modules. The version pin ensures that a new commit to the module repository does not unexpectedly change infrastructure across all consumers. When published to a private registry (Terraform Cloud, Artifactory, or a self-hosted registry), modules appear in a searchable catalog with documentation generated from variables.tf, outputs.tf, and README.md. The registry enforces semantic versioning, making it safe to specify version constraints like ~> 2.3 (any 2.x from 2.3 upward) in consumer configurations.

At production scale, module composition patterns matter as much as individual module quality. The recommended pattern is flat composition: a root configuration references multiple modules at the same level, passing outputs from one module as inputs to another, rather than nesting modules three or four levels deep. Deep nesting creates opaque dependency chains where a change in a leaf module requires understanding the full tree to predict impact. Root modules should be environment-specific (payments-prod, payments-staging) and pin module versions independently per environment so that staging can test a new module version before production adopts it. Teams should run terraform validate and tflint in CI for every module change, and use automated tests with terratest or terraform test to verify module behavior.

The non-obvious gotcha is that module versioning only works if teams actually bump versions. A common failure pattern is pinning to a Git branch (ref=main) instead of a tag, which means terraform init on different days pulls different code. Another trap is overusing count or for_each in modules to make them do too many things — a module that creates either an RDS instance or an Aurora cluster based on a boolean variable becomes untestable and produces confusing plans. Architects should split divergent resources into separate modules rather than adding conditional logic that makes the module's behavior unpredictable.

Think of a highway with automated safety systems. Speed cameras (Sentinel policies) automatically flag violations, fuel cost displays (cost estimation) warn drivers before they commit to a route, roadside inspection stations (run tasks) check specific safety requirements, and GPS-guided lanes (VCS workflows) route each vehicle through the correct path. The highway keeps moving because enforcement is automated, not manual.

Terraform Cloud Enterprise provides a collaborative platform where infrastructure changes follow a standardized workflow: code is committed to VCS, a plan is triggered, governance checks run, and apply executes only after all checks pass. Sentinel is HashiCorp's policy-as-code framework that evaluates Terraform plans, state, and configuration using a policy language. Policies can enforce rules like requiring encryption on all S3 buckets, restricting instance types to cost-approved sizes, mandating specific tags on every resource, or preventing deletion of production databases. Policies are organized into policy sets that are applied to specific workspaces or all workspaces in an organization.

Internally, the run pipeline processes stages in order: VCS trigger, terraform plan, cost estimation, Sentinel policy check, run tasks, and terraform apply. Cost estimation parses the plan output and calculates the monthly cost delta using HashiCorp's pricing database, surfacing changes like adding a db.r6g.2xlarge that increases monthly spend by $1,200. Run tasks are webhook-based integrations that send the plan JSON to external systems — security scanners like Snyk or Prisma Cloud, compliance checkers, or custom approval systems — and wait for a pass/fail response. Each run task can be advisory (warning only) or mandatory (blocking apply). The entire pipeline runs automatically on pull request creation, giving developers feedback in minutes rather than waiting for a manual review.

At production scale, governance design requires balancing safety with velocity. Hard-mandatory Sentinel policies should cover non-negotiable rules like encryption and tagging. Soft-mandatory policies allow overrides with justification for edge cases like temporary large instances for data migration. Advisory policies educate teams about best practices without blocking. Cost estimation thresholds can be set to require manager approval for changes exceeding a dollar amount. VCS workflows should use speculative plans on pull requests (plan only, no apply) so developers see the impact before merging, and auto-apply on the main branch for environments like dev where speed matters more than manual gates.

The non-obvious gotcha is that Sentinel policies execute after the plan phase, so they cannot prevent Terraform from planning invalid configurations — they can only block the apply. If a Sentinel policy references a resource attribute that does not exist in the plan (because the resource was removed), the policy can fail with a confusing error rather than a clean policy violation. Teams should test Sentinel policies against mock plan data in CI using the Sentinel CLI before deploying them to Terraform Cloud. Another trap is over-engineering run tasks: each run task adds latency to the pipeline, and if the external system is slow or unreliable, it blocks every infrastructure change across the organization.

Think of a warehouse inventory system. Drift detection is like a stock audit — you compare what the computer says is on the shelf to what is actually there. Import is like scanning a product that was placed on the shelf without being logged into the system. Move is like changing the shelf label without physically moving the product. All three keep the inventory accurate without throwing anything away.

Infrastructure drift occurs when cloud resources are modified outside Terraform — through the console, CLI, another IaC tool, or automated processes like auto-scaling. terraform plan -refresh-only reads the current state of every managed resource from the cloud provider APIs and compares it to the stored state file. It shows what has changed in the real world without proposing any configuration-level changes. This is distinct from a regular terraform plan, which both refreshes state and compares it to the desired configuration. Running refresh-only plans on a schedule helps teams detect unauthorized changes before they cause incidents.

Internally, refresh-only mode calls the same provider Read functions that a normal plan uses, but it stops after updating the in-memory state representation. It shows a diff between the previously stored state and the freshly read state, highlighting attributes that changed externally. If the operator approves the refresh with terraform apply -refresh-only, the state file is updated to match reality without making any infrastructure changes. Import blocks, introduced in Terraform 1.5, allow declarative imports in configuration files rather than the imperative terraform import CLI command. A resource block with an import block specifies the cloud resource ID, and terraform plan generates the configuration needed to manage it. Moved blocks tell Terraform that a resource has been renamed or restructured in the configuration — for example, moving from a flat resource to a module or changing a resource's for_each key — so it updates the state address rather than planning a destroy and create.

At production scale, drift detection should be automated. Teams run terraform plan -refresh-only in CI on a daily schedule and alert on any detected drift. The plan output is stored as an artifact for audit trail. Import blocks are essential during brownfield adoption — when a company has existing infrastructure created manually or by CloudFormation and wants to manage it with Terraform. Without import blocks, the alternative is terraform import commands that must be run manually for each resource, which is error-prone and not version-controlled. Moved blocks are critical during refactoring: when a team restructures modules, renames resources for clarity, or converts single resources to for_each collections, moved blocks prevent Terraform from destroying the production database and recreating it.

The non-obvious gotcha with refresh-only is that it only detects drift in resources Terraform already manages — it cannot find resources created outside Terraform. Teams need cloud-native tools like AWS Config or Azure Policy for complete drift coverage. With import blocks, the generated configuration may not match the team's coding standards and needs manual cleanup. With moved blocks, the from address must exactly match the current state address, including module paths and index keys, and a typo silently creates a new resource instead of moving the existing one. Architects should always run terraform plan after adding moved blocks and verify that no destroy/create actions appear.

Think of a restaurant chain opening locations in different cities. Rather than writing a completely new business plan for each city, the headquarters uses one standard restaurant blueprint and customizes the local supplier list, health department contact, and rental agreement per location. Provider aliases in Terraform work the same way — one infrastructure blueprint is deployed across regions and accounts by swapping the provider configuration.

In AWS, multi-account architecture is the recommended pattern for blast-radius isolation: production, staging, shared services, security, and logging each live in separate AWS accounts under an AWS Organization. Multi-region deployment adds resilience and latency optimization by placing infrastructure closer to users. Without careful Terraform design, this creates an explosion of near-identical configuration files — one per account-region combination — that diverge over time and become unmaintainable.

Internally, Terraform provider aliases allow declaring multiple instances of the same provider with different configurations. A provider block with alias = "us_west" and region = "us-west-2" coexists with the default provider using region = "us-east-1". Resources and modules reference a specific provider using the provider or providers argument. For cross-account access, each aliased provider uses assume_role to temporarily adopt an IAM role in the target account. The for_each meta-argument on modules, combined with a providers map, enables deploying the same module across multiple regions or accounts from a single configuration. Each module instance receives its own provider, which determines where the infrastructure is created.

At production scale, the pattern involves a locals block that defines a map of regions or account-region pairs, a module block with for_each over that map, and a dynamic providers assignment that passes the correct aliased provider to each module instance. State should be split so that each account-region combination has its own state file — otherwise a single state file becomes a massive blast radius. The deployment pipeline should use separate workspaces or directories per account-region pair, with the provider configuration driven by workspace-specific variables. Teams should also use terraform_remote_state or SSM parameters to share outputs like VPC IDs across account-region boundaries.

The non-obvious gotcha is that Terraform does not support for_each on provider blocks themselves — you cannot dynamically generate provider aliases from a map. Each provider alias must be declared statically in the configuration. This means if you add a new region, you must add a new provider alias block, which is a manual step that cannot be fully automated. Some teams work around this by using Terragrunt to generate provider blocks from a configuration file, or by using a code generator that produces the provider declarations. Another trap is that assume_role credentials expire during long applies — for large deployments, the role session duration must be set high enough (up to 12 hours for chained roles) or the apply will fail midway with an expired token error.

Architecture Layers

Layer 1: Platform Modules (maintained by platform team) - VPC, networking, IAM baseline, logging, security guardrails - Published as versioned modules in a private registry (Terraform Cloud or S3) - Semantic versioning: teams pin to major versions, get patches automatically

Layer 2: Team Modules (self-service wrappers) - Opinionated wrappers around platform modules - Example: team-service module creates ECS service + ALB + DNS + monitoring - Teams provide minimal inputs: service name, container image, CPU/memory - Enforces organizational standards (tagging, encryption, logging)

Layer 3: Team Configurations (per-team repos) - Each team has a repo with their environment definitions - Uses team modules via versioned references - Separate state files per environment (dev/staging/prod)

Imagine a parking garage that assigns spots purely by arrival order — car 1 in spot 1, car 2 in spot 2, and so on. If the third car ever leaves, garage management doesn't leave spot 3 empty; they shove every car behind it forward by one spot to keep things tidy. Every car now has a new spot number, even though most of them never moved their actual car. That's exactly what count does to your infrastructure: the identity of a resource is just its position in a list, not anything meaningful about the resource itself.

Terraform was designed so that each resource in state needs a stable address to track across applies. count.index gives you a cheap way to create N similar resources, and it's fine for genuinely interchangeable things like N identical subnets in a CIDR range. But the moment the members of the list have individual identity — 50 microservices each with their own name, own image, own environment variables — count conflates 'position in the list' with 'identity of the thing,' and Terraform has no way to tell the difference between 'this microservice was removed' and 'everything after index 12 shifted.'

Internally, when you change a list count is iterating over — say removing 'checkout-worker' from position 12 out of 50 — Terraform's plan diff compares the old state addresses (microservice_ecs_service[12] through [49]) against the new ones. Since 'checkout-worker' is gone, positions 13-49 now map to different services than before, so Terraform sees resource[12] changing from checkout-worker's config to notifications-worker's config, and proposes destroying and recreating every single downstream resource just to relabel them — even though 37 of those services never actually changed.

for_each solves this by keying resources with a map, for example for_each = var.microservices where microservices is a map like { "payments-api" = {...}, "checkout-worker" = {...} }. Now each resource address is aws_ecs_service.svc["checkout-worker"], a string key tied to the service's actual name, completely independent of ordering or what else exists in the map. Removing checkout-worker from the map only destroys that one resource; every other service's address, and therefore its state entry, is untouched.

At production scale for 50 microservices specifically, the real answer goes one level further: wrap the for_each block inside a reusable module (module "microservice" { for_each = var.services source = "./modules/ecs-microservice" ... }), so the actual ECS service, task definition, target group, and autoscaling policy definitions live in one place and get parameterized per service via a values map — often sourced from a YAML or JSON file, or even a separate repo per team, so each microservice team owns its own entry without needing to touch the shared root module. The non-obvious gotcha: switching an existing count-based resource to for_each is itself a breaking, destructive change unless you first run terraform state mv for every resource to remap old numeric addresses to the new string-keyed addresses — engineers who skip this step and just swap count for for_each in the code will watch Terraform plan to destroy and recreate everything on the very next apply.