Terraform Advanced Interview Questions

Think of Terraform state like the official city property registry. If two clerks update the same parcel records at the same time, one may overwrite the other's change and the registry becomes untrustworthy. Remote state puts the registry in a shared office instead of on one clerk's laptop, and locking is the sign on the desk that says one clerk is actively editing this registry right now. Without that sign, production changes become a race.

In Terraform, state maps configuration resources to real infrastructure objects and stores attributes Terraform needs for planning. Local state is simple for one person, but it breaks down for teams because every operator needs the newest file and must coordinate manually. Remote backends such as HCP Terraform, S3 with locking, Azure Blob, or GCS move state to a shared system. Locking prevents concurrent plans or applies from corrupting the same state or producing plans from stale assumptions.

The internal flow is: Terraform loads configuration, initializes the backend, reads the latest state, attempts to acquire a lock, refreshes resource data through providers, builds a dependency graph, produces a plan, and applies changes if approved. During apply, state is updated as resources change. If the process crashes, the lock may need cleanup, but force-unlocking should be treated like removing a warning tag from dangerous machinery: only do it after confirming no run is active.

At scale, teams split state to reduce blast radius, improve performance, and align ownership. A network platform state might own VPCs, subnets, and transit gateways, while an application state consumes published subnet IDs. The dangerous split is by file size or folder convenience rather than lifecycle. If an app state creates security groups that the network state mutates, or two states manage different arguments on the same resource, Terraform cannot reason globally and drift becomes normal.

The non-obvious gotcha is that remote state outputs are not a service discovery system. They expose snapshots from another state, and consumers may act on outputs that are valid syntactically but operationally stale. Senior engineers prefer stable contracts, narrow outputs, versioned modules, explicit ownership, and separate data stores for values consumed by non-Terraform systems. They also design emergency unlock procedures, backend access controls, and audit trails before the first production incident.

Terraform state locking is a concurrency control mechanism that prevents two or more operators from writing to the same state file simultaneously, which would cause state corruption and potentially orphaned cloud resources. Think of it like a database row-level lock: before Terraform can modify state, it must first acquire an exclusive lock, and any other process attempting to modify the same state must wait or fail.

When you run terraform apply or terraform plan (with certain backends), Terraform sends a Lock request to the backend. For S3+DynamoDB, this means writing a record to the DynamoDB table with the state file's digest as the partition key and a unique LockID. The LockID contains the operator's hostname, the Terraform operation type, the workspace name, and a timestamp. DynamoDB's conditional write ensures atomicity: if a record already exists with that key, the write fails with a ConditionalCheckFailedException, and Terraform reports that the state is locked by another process.

In production, lock conflicts arise in several scenarios. The most common is when two engineers run terraform apply on the same workspace concurrently. Terraform will display the lock holder's information including their username, operation type, and when the lock was acquired. The blocked operator must wait for the first operation to complete. Another common scenario is a CI/CD pipeline crash: if a pipeline runner dies mid-apply, the lock remains in DynamoDB as a stale lock. This requires manual intervention using terraform force-unlock with the lock ID.

Force-unlock is dangerous in production because you cannot guarantee the previous operation completed cleanly. Before force-unlocking, you should verify the state of the actual infrastructure using AWS console or CLI, check the DynamoDB table directly to confirm the lock metadata, and review CloudTrail logs for any API calls made by the crashed process. A safer pattern is to implement lock timeouts in your CI/CD pipeline: wrap terraform apply in a timeout command and have the pipeline explicitly run terraform force-unlock only after confirming no infrastructure changes are in progress.

Different backends handle locking differently. Consul uses its built-in session-based locking with TTL. Azure Blob Storage uses native blob leases. Google Cloud Storage uses object generation numbers for optimistic locking. Not all backends support locking; the local backend does via filesystem locks, but NFS-mounted local backends have notoriously unreliable locking, which is why teams migrate to remote backends. The etcd backend uses its compare-and-swap primitive. Understanding your backend's locking semantics is critical for disaster recovery planning, because a corrupted lock table can block all infrastructure changes across your organization.

Structuring Terraform for a multi-account AWS organization requires solving three interconnected problems: provider authentication across accounts, state isolation between accounts, and code reuse across similar environments. Think of it like managing a franchise: each store (account) runs the same playbook but has its own inventory (state), and headquarters (management account) needs oversight into all of them.

The foundation is AWS Organizations with a well-defined account structure. Typically you have a management account (for Organizations API and billing), a security account (for GuardDuty, SecurityHub, CloudTrail aggregation), a shared-services account (for CI/CD, artifact repositories, DNS), and then workload accounts per environment or per team. Each account gets its own Terraform state file to ensure blast radius isolation: a misconfigured apply in the staging account cannot corrupt production state.

For provider configuration, the recommended pattern is assume-role chaining. Your CI/CD pipeline authenticates to a central deployment account using OIDC (for GitHub Actions) or instance profiles (for Jenkins on EC2), then assumes environment-specific roles in target accounts. Each account has a TerraformExecutionRole with least-privilege permissions. The provider block uses assume_role with the target account's role ARN, and you parameterize the account ID using variables or a map lookup.

The repository structure typically follows one of two patterns. The first is a monorepo with directory-per-account: infrastructure/accounts/production/, infrastructure/accounts/staging/, each with their own backend configuration and tfvars. The second is a module-based approach where a single root configuration uses for_each over a map of accounts to deploy baseline resources. The monorepo approach is simpler to reason about but leads to code duplication. The module approach is DRY but increases blast radius.

Shared modules are the cornerstone of multi-account management. You create versioned modules for common patterns: a VPC module that enforces CIDR allocation from a central IPAM, a security-baseline module that deploys Config rules and GuardDuty, an IAM-baseline module that creates standard roles. These modules are published to a private Terraform registry or referenced via Git tags.

Production gotchas are numerous. Cross-account resource references require careful handling: you cannot reference a resource in another account's state without remote state data sources or SSM Parameter Store lookups. VPC peering across accounts needs accepter-side resources managed by the accepter account's Terraform. Service Control Policies in the management account can block Terraform operations in child accounts if not carefully scoped. And state backend permissions must be tightly controlled: each account's Terraform role should only access its own state prefix in S3.

Terraform moved blocks, introduced in Terraform 1.1, provide a declarative way to refactor your configuration without destroying and recreating real infrastructure. Think of it like a postal forwarding address: you tell Terraform that the resource formerly known as aws_instance.web_server now lives at module.compute.aws_instance.payments_api, and Terraform updates its internal address book (the state file) without touching the actual mail (cloud resources).

Before moved blocks, refactoring Terraform code was perilous. If you renamed a resource from aws_rds_instance.database to aws_rds_instance.payments_db, Terraform would plan to destroy the old resource and create a new one, because it tracks resources by their configuration address. The workaround was terraform state mv, an imperative command that modifies state directly. This was error-prone, not version-controlled, and required coordination: every team member had to run the state mv command in the right order before applying, or their plan would show destructive changes.

Moved blocks solve this elegantly. You add a moved block to your configuration that declares the old and new addresses. When Terraform processes the configuration, it reads the moved blocks before computing the plan, updates the state mapping in memory, and then generates a plan based on the corrected addresses. The result is a plan that shows no changes (or only the changes you intentionally made), and the state file is updated as part of the normal apply process.

Moved blocks support several refactoring patterns. Resource renaming is the simplest: changing a resource's logical name. Module restructuring lets you move resources into or out of modules, or between modules. Adding for_each to a resource that was previously a single instance requires moving from the singleton address to an indexed address. You can also use moved blocks when splitting a monolithic configuration into smaller modules.

Production gotchas are important to understand. Moved blocks are processed in order, so if you have a chain of moves (A to B, then B to C), they must be listed in sequence. You cannot move a resource across provider boundaries: moving aws_instance to google_compute_instance is not valid because the underlying APIs and schemas differ. Moved blocks should be kept in configuration temporarily during the transition period and removed after all workspaces have applied the change, because stale moved blocks referencing non-existent source addresses will generate warnings. When moving resources that have count or for_each, you must specify the exact index or key in both the from and to addresses. Finally, moved blocks do not work across state file boundaries: you cannot move a resource from one state to another using moved blocks alone; that still requires terraform state mv with the -state flags or import blocks.

Terraform plan's drift detection works through a refresh-then-diff process. Think of it like an inventory audit: Terraform reads the last known inventory (state file), physically checks every item in the warehouse (API calls to cloud providers), updates the inventory with actual findings (state refresh), and then compares the updated inventory against the blueprint (configuration). Any discrepancies between the refreshed state and the desired configuration become the plan.

The refresh phase is where drift detection happens. For every resource tracked in the state file, Terraform calls the provider's ReadResource RPC method, which translates to cloud API calls. For an aws_rds_cluster.payments_db, this triggers a DescribeDBClusters API call. The provider compares the API response against the state file's recorded attributes. If the production database's backup_retention_period was changed from 30 to 7 via the AWS console, the refresh detects this as drift and updates the in-memory state.

After refresh, Terraform diffs the refreshed state against the configuration. If your configuration says backup_retention_period = 30 but the refreshed state shows 7, the plan proposes changing it back to 30. This is Terraform's self-healing property: it converges actual infrastructure toward the declared configuration.

However, the limitations are significant and often misunderstood in production. First, Terraform only detects drift on resources it manages. If someone creates an additional security group rule via the AWS console that is not in Terraform's state, Terraform has no knowledge of it. This is the 'unknown unknowns' problem: Terraform cannot detect resources it does not track.

Second, not all providers report all attributes during refresh. Some cloud APIs return partial data, or certain attributes are write-only (like passwords). The AWS provider, for example, cannot detect drift on certain IAM policy document orderings because the API returns a canonicalized version that may not match the original.

Third, the refresh phase can be slow and expensive. In a large infrastructure with thousands of resources, the refresh makes thousands of API calls, which can hit rate limits and take tens of minutes. Terraform 1.5 introduced the -refresh=false flag to skip refresh for faster plans, but this trades drift detection for speed.

Fourth, eventual consistency in cloud APIs can cause false drift detection. After an AWS resource is created, the API may return stale data for seconds or minutes. Running plan immediately after apply can show phantom drift that resolves itself.

Fifth, Terraform cannot detect drift on resource dependencies that are not explicitly modeled. If a VPC peering connection's route table was modified outside Terraform but the peering resource itself was not, Terraform might not detect the functional impact. Tools like AWS Config, CloudTrail-based drift detection, or Driftctl (now part of Snyk) fill these gaps by scanning entire accounts for unmanaged resources.

A production-grade Terraform CI/CD pipeline must solve five problems: authentication without long-lived credentials, plan visibility for reviewers, approval gates before destructive changes, concurrency control to prevent conflicting applies, and policy enforcement to catch compliance violations before they reach infrastructure. Think of it like a surgical operation workflow: the surgeon (engineer) proposes an operation (plan), it gets reviewed by a board (PR review), approved by an authority (manual approval gate), and only then executed in a controlled environment (apply) with safeguards (state locking).

Authentication should use OIDC federation. GitHub Actions, GitLab CI, and CircleCI all support OIDC tokens that can be exchanged for short-lived AWS credentials via STS AssumeRoleWithWebIdentity. This eliminates the need to store AWS access keys as CI secrets, which is a common audit finding. The OIDC trust policy should be scoped to specific repositories and branches to prevent unauthorized access.

The pipeline structure typically has three stages. The first stage runs on every pull request: terraform init, terraform validate, terraform fmt -check, and terraform plan -out=plan.tfplan. The plan output is captured and posted as a PR comment using a tool like tfcmt or a custom script that parses the plan JSON output. This gives reviewers visibility into exactly what will change.

The second stage is the approval gate. For non-production environments, this might be automatic after PR merge. For production, it requires explicit manual approval. In GitHub Actions, this is implemented using environments with required reviewers. In GitLab, it is a manual job gate. The approval should be from someone other than the PR author (four-eyes principle) and ideally from a platform engineering team member who understands the blast radius.

The third stage runs terraform apply using the saved plan file. This is critical: never re-run plan during apply, because infrastructure may have changed between the plan and apply stages. The saved plan file ensures exactly what was reviewed gets applied. After apply, the pipeline should post the apply output back to the PR or a Slack channel for visibility.

Policy-as-code adds guardrails. HashiCorp Sentinel (Terraform Cloud/Enterprise) or Open Policy Agent (open source) evaluate the plan against organizational policies: no public S3 buckets, all RDS instances must have encryption, all security groups must have descriptions. These checks run after plan but before approval, catching violations early.

Production gotchas include handling plan file expiration (plan files reference specific provider plugin versions and state serial numbers, so they expire when state changes), managing workspace-level parallelism (only one pipeline should operate on a workspace at a time), and dealing with long-running applies that exceed CI timeout limits. Some teams implement a Terraform-specific lock in Redis or DynamoDB beyond the state lock, to queue pipeline runs at the workspace level.

Terraform's provider architecture is built on HashiCorp's go-plugin framework, which uses gRPC for inter-process communication. Think of it like a restaurant: Terraform core is the head chef who designs the menu (configuration) and coordinates the kitchen, while providers are specialized line cooks (AWS, Azure, GCP) who know how to prepare specific dishes (resources). They communicate via tickets (gRPC messages), and each cook can be replaced or upgraded without retraining the head chef.

When you run terraform init, Terraform downloads provider binaries from the configured registry (registry.terraform.io by default) and stores them in the .terraform/providers directory. Each provider is a standalone Go binary compiled for your platform. When Terraform needs to interact with a provider, it launches the binary as a subprocess. The provider binary starts a gRPC server on a random local port and communicates the port back to Terraform core via stdout using a handshake protocol. This is the 'plugin negotiation' phase.

The gRPC protocol defines several key RPCs. GetProviderSchema returns the provider's complete schema: all resource types, data sources, and their attributes with types and validation rules. This is how Terraform knows what arguments aws_rds_cluster accepts without hardcoding AWS-specific knowledge. ValidateResourceConfig checks that user-provided configuration is structurally valid before any API calls. PlanResourceChange computes the proposed new state for a resource given the current state and desired configuration. ApplyResourceChange executes the actual infrastructure change by calling cloud APIs. ReadResource fetches the current state of a resource for drift detection.

The protocol uses Protocol Buffers for serialization, with a schema called tfplugin6.proto (for protocol version 6, used by modern providers). Resource attribute values are encoded as MessagePack within the protobuf messages, using a type system called cty (a Go library for dynamic typing). This double encoding exists because protobuf alone cannot express Terraform's complex type system (nested blocks, sets of objects, sensitive values).

Provider development uses the Terraform Plugin Framework (the modern approach) or the older Terraform Plugin SDK v2. The framework provides abstractions for implementing the gRPC service methods. When you write a provider resource, you implement methods like Create, Read, Update, Delete, and the framework translates these into the appropriate gRPC responses.

Production implications of this architecture include: provider version pinning is critical because different provider versions expose different gRPC schemas, and a schema mismatch can corrupt state. The subprocess model means provider crashes do not crash Terraform core, but they can leave resources in an inconsistent state. Provider parallelism is controlled by the -parallelism flag, which limits how many concurrent RPC calls Terraform makes. Network-isolated environments need a provider mirror or filesystem mirror because init downloads from the internet by default. Understanding this architecture is essential for debugging provider-specific errors: when Terraform reports 'provider produced inconsistent result,' it means the provider's ApplyResourceChange returned state that does not match its PlanResourceChange prediction.

Testing Terraform code requires a layered approach because infrastructure code has fundamentally different risk profiles than application code. Think of it like testing a building design: you have blueprint reviews (static analysis), structural simulations (plan-based testing), scale model tests (integration tests in isolated accounts), and the final building inspection (end-to-end tests against real infrastructure).

The first layer is static analysis, which catches errors without any provider interaction. terraform validate checks HCL syntax and basic type errors. terraform fmt -check enforces consistent formatting. tflint goes further with provider-aware linting: it can detect invalid instance types (aws_instance with instance_type = 'm5.huuge'), deprecated resource arguments, and naming convention violations. Custom tflint rules can enforce organizational standards. Checkov and tfsec scan for security misconfigurations: public S3 buckets, unencrypted databases, overly permissive security groups. These tools run in seconds and belong in pre-commit hooks and CI pipeline early stages.

The second layer is plan-based testing, which runs terraform plan and asserts on the planned changes without actually modifying infrastructure. The native terraform test framework (introduced in Terraform 1.6) allows writing tests in HCL files with .tftest.hcl extension. Each test file defines variables, runs plan or apply commands, and asserts on outputs or resource attributes. Plan-mode tests are fast and safe: they verify that your configuration produces the expected plan without touching real infrastructure. You can assert that specific resources will be created, that computed values match expectations, and that no unexpected destroys are planned.

The third layer is integration testing with terraform test in apply mode or with Terratest. Terraform test in apply mode provisions real infrastructure in an isolated test account, runs assertions against the live resources, and destroys everything afterward. Terratest (a Go library by Gruntwork) provides more flexibility: you can make HTTP requests to deployed endpoints, run SQL queries against provisioned databases, SSH into EC2 instances to verify configuration, and use retry logic for eventually consistent resources.

Terratest's power comes from Go's testing ecosystem. You write Go test functions that call terraform.InitAndApply, then use AWS SDK calls to validate the infrastructure matches expectations. For example, after deploying an RDS cluster, you can use the AWS SDK to verify encryption is enabled, multi-AZ is configured, and the parameter group has the expected settings. The test function calls terraform.Destroy in a deferred cleanup block.

Production gotchas for Terraform testing include: test account isolation is critical because integration tests create real resources with real costs. Use AWS Organizations with a dedicated testing account and service control policies to prevent tests from accessing production. Implement cost controls with AWS Budgets on test accounts. Parallelize tests carefully because Terraform state conflicts can occur if multiple test runs target the same state backend. Use unique naming prefixes or workspaces per test run. Terraform test has limitations with provider mocking (experimental in 1.7) and cannot easily test cross-module interactions. Terratest requires Go proficiency and has slower test execution times due to real infrastructure provisioning. A balanced strategy uses static analysis in pre-commit hooks, plan assertions in CI, and full integration tests nightly or before releases.

Terraform state corruption is one of the most dangerous operational incidents because the state file is the single source of truth mapping your configuration to real infrastructure. Think of it like a hospital's patient registry: if the registry is corrupted, you do not know which patient is in which room, and any action you take risks harming the wrong patient. State corruption can cause Terraform to destroy resources it thinks do not exist, create duplicates of resources it cannot find, or fail entirely with cryptic deserialization errors.

State corruption occurs in several ways. The most common is a partial write during terraform apply: if the process is killed mid-apply (OOM killer, network interruption, CI timeout), the state file may contain a partially updated resource. Another common cause is manual state file editing with JSON syntax errors. Concurrent writes without proper locking can interleave state updates. Provider bugs can write malformed attribute values. And rarely, S3 eventual consistency (in older S3 behavior) could serve a stale state version.

The first line of defense is prevention. Enable S3 versioning on your state bucket so every state write creates a new version, giving you point-in-time recovery. Enable MFA delete on the bucket to prevent accidental or malicious version deletion. Use DynamoDB state locking to prevent concurrent writes. Run terraform plan before apply to catch state inconsistencies early. Implement CI/CD pipelines that prevent direct state manipulation.

When corruption occurs, follow a structured recovery procedure. First, immediately stop all Terraform operations on the affected workspace. If your CI/CD pipeline has queued runs, cancel them. This prevents compounding the corruption.

Second, assess the damage. Run terraform state pull to download the current state file and inspect it. Check the JSON structure: is it valid JSON? Are the serial number and lineage fields present? Use jq to examine specific resources. Compare the state against your actual infrastructure using AWS CLI or console.

Third, attempt recovery from backup. If S3 versioning is enabled, list previous versions with aws s3api list-object-versions, download a known-good version, and push it back using terraform state push. The state push command validates the state format and updates the serial number. Be careful with the -force flag: it skips lineage checking, which is a safety mechanism that prevents pushing state from the wrong workspace.

Fourth, if no backup is available, perform selective state surgery. Use terraform state rm to remove the corrupted resource entries, then terraform import to re-import the existing infrastructure resources. This is tedious for large configurations but preserves the non-corrupted portions of state. For each imported resource, run terraform plan to verify the import produced a state entry that matches your configuration.

Fifth, as a last resort for total state loss, you can rebuild the entire state from scratch using terraform import for every resource. This is the infrastructure equivalent of a bare-metal restore. Tools like terraformer and former2 can help by scanning your AWS account and generating import commands, but they require careful validation.

After recovery, conduct a post-mortem. Implement additional safeguards: S3 bucket replication to a separate account for disaster recovery, automated state backup jobs that copy state to a different storage system, monitoring on state file size and serial number for anomaly detection, and regular recovery drills where you practice restoring state from backup in a non-production workspace.

Think of importing infrastructure like adding existing buildings to a city map. You need the right address, the right owner, and the right description; otherwise future construction permits target the wrong building.

Terraform import exists so unmanaged resources can come under infrastructure-as-code control. Modern import blocks make that workflow declarative, reviewable, and repeatable instead of a one-off terminal action.

The workflow is to identify resource identity, declare an import block with the destination address, create or generate a matching resource block, run plan, then apply. Terraform records the remote object in state at the declared address.

In production, risk lives in the address and configuration match. If attributes do not match real infrastructure, the first post-import plan may propose dangerous changes. Engineers check lifecycle rules, tags, provider aliases, and module paths carefully.

The non-obvious gotcha is assuming import means Terraform understands the whole resource relationship. It only records the target object; you still need to model dependencies, ownership, and ignored provider-managed fields correctly.

Deploying EKS with Terraform is like building a three-story building: the foundation is networking (VPC, subnets, NAT gateways), the structural frame is the EKS control plane (API server, etcd, OIDC provider), and the floors are the node groups (compute capacity where workloads actually run). Each layer depends on the one below it, and you should be able to rebuild any floor without demolishing the entire building.

The networking module provisions a dedicated VPC with public subnets for load balancers, private subnets for worker nodes, and optionally isolated subnets for databases. EKS requires specific subnet tags: kubernetes.io/cluster/<cluster-name> = shared on all subnets, kubernetes.io/role/elb = 1 on public subnets for internet-facing ALBs, and kubernetes.io/role/internal-elb = 1 on private subnets for internal services. The module outputs subnet IDs and the VPC ID for consumption by the cluster module. NAT gateways should be deployed per-AZ in production for high availability, meaning three NAT gateways across us-east-1a, us-east-1b, and us-east-1c.

The cluster module creates the EKS control plane using the aws_eks_cluster resource or the terraform-aws-modules/eks/aws community module. Critical configurations include the Kubernetes version (pin to a specific minor version like 1.29), the cluster endpoint access (private-only or public-and-private with CIDR restrictions), envelope encryption for secrets using a dedicated KMS key, and the OIDC provider for IAM Roles for Service Accounts (IRSA). The OIDC provider is frequently missed but essential: it enables pods to assume IAM roles without injecting AWS credentials, which is the only secure way to grant AWS access to workloads.

The node groups module manages EKS managed node groups with launch templates. Production clusters typically need multiple node groups: a system node group (t3.xlarge, 3 nodes, taints for system workloads like CoreDNS and kube-proxy), an application node group (m5.2xlarge, 3-15 nodes with cluster autoscaler), and optionally a GPU node group (g4dn.xlarge for ML inference). Each node group uses a custom launch template to specify the AMI (Amazon EKS-optimized AMI), bootstrap arguments, block device mappings (100Gi gp3 root volume), and user data for kubelet configuration. Instance refresh policies ensure rolling updates when the launch template changes.

In production, state separation between these modules is critical. If your node group Terraform runs into an error, you do not want it to affect the EKS control plane state. Use separate state files: one for networking, one for the cluster, and one per node group pool. Pass data between modules using terraform_remote_state data sources or aws_ssm_parameter lookups. This blast radius isolation means a botched node group change cannot accidentally destroy the control plane.

A common gotcha is the chicken-and-egg problem with EKS add-ons. The aws-auth ConfigMap (which controls IAM-to-Kubernetes RBAC mapping) requires a running cluster, but node groups need the aws-auth ConfigMap to join the cluster. The solution is to use the EKS access entries API (available since EKS platform version eks.8) instead of managing aws-auth directly, or to use the kubernetes provider with the EKS cluster's endpoint and token to manage the ConfigMap in the same apply as the cluster creation.

Structuring AWS accounts for multiple environments is like managing a hospital: you would never put the emergency room (production), the training lab (dev), the simulation center (QA), and the dress rehearsal ward (UAT) in the same building with shared keys and power circuits. AWS Organizations provides the building-per-department model, giving each environment complete blast radius isolation at the account boundary.

The multi-account structure typically follows an organizational unit (OU) hierarchy. The root OU contains the management account (billing and Organizations API only — never deploy workloads here). Below it, you create OUs for Security (GuardDuty delegated admin, SecurityHub, CloudTrail aggregation), SharedServices (CI/CD runners, ECR registry, Route53 hosted zones, Terraform state bucket), and Workloads. The Workloads OU contains sub-OUs for NonProd (Dev, QA, UAT accounts) and Prod (production account). Service Control Policies (SCPs) at each OU level enforce guardrails: NonProd accounts cannot provision p4d.24xlarge instances or create public-facing resources, while the Prod OU has SCPs preventing deletion of CloudTrail logs or disabling encryption.

In Terraform, each account is targeted through provider assume_role blocks. The CI/CD pipeline authenticates to the SharedServices account via OIDC, then assumes a TerraformExecutionRole in the target account. This role is provisioned by an account-baseline module that every new account receives. The role has permissions scoped to the services that environment needs — Dev gets broad permissions for experimentation, while Prod has tightly scoped permissions with explicit deny on destructive actions like deleting RDS clusters without snapshots.

State file organization follows the account boundary. A single S3 bucket in the SharedServices account stores all state files, with key prefixes per account: s3://org-terraform-state/111111111111/networking/terraform.tfstate for the Dev account networking stack, s3://org-terraform-state/222222222222/networking/terraform.tfstate for Prod. Each account's TerraformExecutionRole has an S3 policy that restricts access to only its own prefix, preventing a Dev pipeline misconfiguration from reading or writing Prod state.

The single-account approach — using naming conventions and tags to separate environments — is tempting for small teams but creates dangerous failure modes at scale. A single IAM policy mistake can grant Dev workloads access to Prod databases. Security groups in a shared VPC can be referenced across environments. Billing attribution becomes guesswork with cost allocation tags instead of per-account bills. Most critically, AWS service quotas are shared: a runaway Dev autoscaling group can exhaust EC2 limits and prevent Prod from scaling during a traffic spike.

The gotcha with multi-account is cross-account resource sharing. VPC peering or Transit Gateway connects environments for legitimate data flows (QA reading anonymized Prod data, shared ECR images). Terraform must manage both sides of a peering connection: the requester in one account and the accepter in another, each with their own provider alias. This requires careful orchestration — apply the requester first, then the accepter — or use a two-phase apply with data sources that look up the peering connection ID.

Terraform state file management is like managing a bank vault's ledger: the ledger (state file) records what is in every safe deposit box (cloud resource), and if the ledger is corrupted or leaked, you either lose track of assets or expose their locations to unauthorized parties. The storage, encryption, access control, and corruption prevention for state files must be treated with the same rigor as production database backups.

The S3 backend is the standard for AWS-centric teams. The bucket itself requires several hardening measures: versioning enabled so you can recover previous state versions if an apply corrupts the current state, server-side encryption with a dedicated KMS key (not the default aws/s3 key) so you can audit and rotate encryption independently, public access blocked via the S3 Block Public Access settings, and a bucket policy that explicitly denies unencrypted uploads. The bucket should be in a dedicated SharedServices or Management account, separate from any workload account, so that workload account compromises cannot directly access state.

The key structure within the bucket follows a hierarchy: {account-id-or-env}/{stack-name}/terraform.tfstate. For example: prod/networking/terraform.tfstate, prod/eks-cluster/terraform.tfstate, prod/payments-database/terraform.tfstate. This structure enables per-stack state isolation and per-environment IAM scoping. The Prod account's TerraformExecutionRole gets an S3 policy allowing s3:GetObject and s3:PutObject only on keys prefixed with prod/, while the Dev role can only access dev/. This prevents a Dev pipeline misconfiguration from overwriting Prod state.

DynamoDB state locking prevents concurrent modifications. Create a single DynamoDB table (PAY_PER_REQUEST billing) with a partition key named LockID of type String. Every Terraform operation acquires a lock before modifying state by writing a conditional item to this table. If two engineers run terraform apply simultaneously on the same stack, the second operation receives a ConditionalCheckFailedException and waits. The lock record contains the operator's hostname, the operation type, and a timestamp, which helps diagnose stale locks from crashed CI pipelines.

Corruption prevention goes beyond locking. S3 versioning provides a recovery path: if an apply fails midway and leaves state inconsistent, you can restore a previous version using aws s3api list-object-versions and aws s3api get-object with the desired VersionId. Terraform also writes a backup of the previous state locally before modifying it (terraform.tfstate.backup), though this is less useful in CI/CD where runners are ephemeral. For critical production stacks, enable S3 Replication to copy state to a bucket in another region for disaster recovery.

The most dangerous corruption scenario is partial apply failure: Terraform creates some resources but crashes before writing updated state. The created resources become orphans — they exist in AWS but are not tracked by Terraform. Recovery requires manually importing the orphaned resources using terraform import or, in Terraform 1.5+, using import blocks. To reduce this risk, break large configurations into smaller stacks so each apply touches fewer resources, and use the -target flag only as a last resort since it creates partial state updates by design.

Preventing cross-environment contamination in Terraform is like building firewalls between apartments in a building: you need physical separation (state isolation), locked doors (IAM boundaries), independent utilities (provider configurations), and a building code (CI/CD guardrails) that prevents shortcuts through shared walls.

The first layer is state file isolation. Each environment must have its own state file with its own backend configuration. Never share a state file between environments, even with workspaces, if the blast radius of corruption is unacceptable. The state file contains sensitive data including resource IDs, IP addresses, and sometimes plaintext outputs. An S3 bucket policy should restrict each environment's Terraform role to only its own key prefix: the prod role can access s3://state-bucket/prod/* but is explicitly denied s3://state-bucket/dev/*. This prevents a misconfigured prod pipeline from reading or overwriting dev state.

The second layer is provider-level isolation. Each environment's provider block must assume a role in its specific AWS account. Even if someone accidentally passes the wrong tfvars file, the provider configuration ensures Terraform operates in the correct account. Add a validation check using the aws_caller_identity data source: compare the actual account ID against the expected one and fail early if they do not match. This catches the scenario where an engineer runs terraform apply with prod credentials but dev configuration, or vice versa.

The third layer is module versioning. When environments share modules from a private registry or Git repository, use pinned version tags. Dev might use module version 2.3.0-rc1 while Prod uses 2.2.0 (the last stable release). Without version pinning, a module change pushed to the main branch immediately affects every environment that references source = "git::...?ref=main". This is the most common cause of accidental cross-environment impact: someone fixes a bug in a shared VPC module, the fix has a typo, and every environment that references the module head picks up the broken code on next apply.

The fourth layer is CI/CD pipeline guardrails. The pipeline should validate environment consistency before plan: check that the workspace name matches the tfvars file, verify the AWS account ID matches the target environment, and confirm the Git branch is allowed to deploy to that environment (only main can deploy to prod). Implement a pre-plan script that runs aws sts get-caller-identity and compares the account against an expected value from the pipeline configuration.

Remote state data sources are a particularly dangerous vector for cross-environment bleed. When a production EKS module reads the networking module's state via terraform_remote_state, it must reference the production networking state, not dev. Parameterize the remote state data source's backend configuration using the environment variable: data.terraform_remote_state.networking.config.key should resolve to prod/networking/terraform.tfstate, not a hardcoded path. A common gotcha is using terraform_remote_state with a hardcoded key that works in dev but points to prod state when someone copies the configuration without updating the key.

The ultimate safeguard is defense in depth: even if one layer fails, the others prevent damage. If the IAM policy has a bug that allows dev access to prod state, the provider's assume_role still locks operations to the dev account. If the provider configuration is wrong, the account ID validation check fails before any resources are touched.

A Terraform CI/CD pipeline is like an air traffic control system: every infrastructure change (flight) must file a plan (flight plan), get reviewed by controllers (PR reviewers), receive clearance (approval gate), and land on the correct runway (target environment) — all while preventing two planes from using the same runway simultaneously (state locking and concurrency control).

The pipeline begins with authentication. Modern pipelines use OIDC federation instead of stored AWS credentials. GitHub Actions requests a JWT token from GitHub's OIDC provider, presents it to AWS STS via AssumeRoleWithWebIdentity, and receives short-lived credentials scoped to the Terraform execution role. The OIDC trust policy restricts which repositories, branches, and environments can assume the role: production apply roles should only be assumable by the main branch, while plan roles can be assumed by any branch. This eliminates long-lived access keys that could be exfiltrated from CI secrets.

The plan stage runs on every pull request. It executes terraform init, terraform validate, terraform fmt -check, and terraform plan -out=plan.tfplan. The plan output is captured and posted as a PR comment using tools like tfcmt or the native GitHub Actions Terraform setup action. Reviewers see exactly what resources will be created, modified, or destroyed — including sensitive changes like security group rule modifications or IAM policy updates. The saved plan file is uploaded as a CI artifact for use in the apply stage.

Policy-as-code gates run between plan and approval. Open Policy Agent (OPA) evaluates the plan JSON (terraform show -json plan.tfplan) against organizational policies: no S3 buckets without encryption, no security groups with 0.0.0.0/0 ingress on port 22, all RDS instances must have deletion protection in production. These checks are non-negotiable — a policy violation fails the pipeline regardless of who approves the PR. Sentinel serves the same purpose in Terraform Cloud/Enterprise environments.

The approval gate differs by environment. Dev and QA may auto-apply on merge — the PR review itself is sufficient approval. UAT requires team lead approval via a GitHub Environment with one required reviewer. Production requires two approvals from the platform-admins team, with a 15-minute wait timer to prevent hasty approvals. These are configured as GitHub Environments with protection rules, which the apply job references via the environment keyword.

The apply stage triggers after merge to main. Critically, it should use the saved plan file from the plan stage rather than re-running plan, because infrastructure may have changed between plan review and apply execution. If the saved plan is stale (state serial mismatch), Terraform rejects it and the pipeline must re-plan. After successful apply, the pipeline posts results to a Slack channel (#infra-changes-prod) and creates a GitHub deployment record for audit trail.

Concurrency control prevents two merged PRs from applying simultaneously to the same stack. GitHub Actions concurrency groups scoped to the stack name (concurrency: group: terraform-payments-prod) ensure only one apply runs at a time. Queued runs wait for the current apply to complete. Combined with DynamoDB state locking, this provides two layers of concurrent modification prevention.

Terraform state drift is like someone rearranging furniture in a room that has a blueprint: the blueprint (state file) says the couch is by the window, but someone physically moved it to the center of the room (console change). Terraform detects this discrepancy during the refresh phase of plan and proposes moving the couch back to the window (converging to declared state). The question is whether the move was intentional or accidental — and that determines whether you update the blueprint or move the couch back.

Drift detection happens during the refresh phase of terraform plan. For every resource tracked in state, Terraform calls the cloud provider's API to read the current configuration. If the API response differs from what state records, Terraform updates its in-memory state and then diffs that against your HCL configuration. The -refresh-only flag runs only the refresh phase without proposing configuration-driven changes, making it a pure drift detection scan. The output shows which attributes have drifted and their before/after values.

There are three categories of drift, each requiring a different remediation strategy. The first is accidental drift: an engineer manually opened port 443 on a security group to debug a connectivity issue and forgot to revert it. The fix is to run terraform apply, which converges the security group back to the declared configuration, removing the manually added rule. This is Terraform's self-healing property — the declared state is the source of truth.

The second is intentional drift: an operations engineer manually scaled up an RDS instance from db.r6g.xlarge to db.r6g.2xlarge during a traffic incident. The change was correct and should be preserved. The fix is to update the Terraform code to reflect the new instance class, then run terraform plan to verify the plan shows no changes (the code now matches reality). If you run apply without updating the code, Terraform would downgrade the instance back to the original size — potentially causing another outage.

The third is untracked resource creation: someone created a new S3 bucket via the console that Terraform knows nothing about. Since Terraform only tracks resources in its state, it cannot detect untracked resources. Tools like AWS Config, Driftctl (now Snyk IaC), or CloudQuery scan the entire account and compare against Terraform state to find resources that exist but are not managed. Once identified, you either import the resource into Terraform using import blocks (Terraform 1.5+) or the terraform import command, or you delete the resource if it should not exist.

Proactive drift prevention is better than reactive detection. Implement AWS Config rules that alert on configuration changes not made by the Terraform execution role. Set up CloudTrail-based alarms that trigger when console users modify resources tagged with ManagedBy=terraform. Use IAM policies that restrict console users to read-only access for Terraform-managed resource types. Schedule a daily terraform plan -refresh-only in CI that posts drift reports to a Slack channel — this catches drift within 24 hours instead of discovering it during the next deployment.

The lifecycle meta-argument ignore_changes is the escape hatch for expected drift. Auto-scaling groups change desired_capacity based on scaling policies, ECS services change task_count, and some resources have attributes that are set once and then managed externally. Adding these attributes to ignore_changes tells Terraform to skip them during drift comparison, preventing false positives and accidental reverts of legitimate operational changes.

Think of building a house. Terraform is the general contractor who builds the structure — foundation, walls, roof, plumbing, and electrical. Ansible is the interior designer who makes it livable — painting walls, installing appliances, setting up the home network, and arranging furniture. You would never ask the interior designer to pour concrete, and you would not ask the general contractor to pick curtain colors. Each tool excels in its domain, and trying to force one to do the other's job creates fragile, unmaintainable automation.

Terraform excels at declarative infrastructure provisioning through cloud provider APIs. It manages the lifecycle of resources — creating, updating, and destroying VPCs, subnets, security groups, EC2 instances, RDS databases, EKS clusters, S3 buckets, and IAM roles. Terraform's state file tracks what exists and what needs to change, enabling plan-before-apply workflows that show exactly what will be created, modified, or destroyed. For banking infrastructure, Terraform manages the network topology (hub-and-spoke VPC with transit gateway), the compute layer (EKS clusters, EC2 bastion hosts), the data layer (RDS PostgreSQL for settlements-db, ElastiCache for session management), and the security layer (KMS keys, WAF rules, Security Hub integration). Terraform is idempotent through its state — running apply twice produces no changes if the infrastructure matches the desired state.

Ansible excels at procedural configuration management through SSH or WinRM. It manages what happens inside servers — installing packages, configuring services, managing files, hardening operating systems, and deploying applications to non-containerized targets. For banking, Ansible handles CIS benchmark hardening on EC2 instances, installing and configuring monitoring agents (Datadog, Splunk forwarder), managing SSL certificates on legacy application servers, configuring LDAP authentication, and deploying Java applications to Tomcat servers that have not yet been containerized. Ansible is idempotent through its modules — each module checks the current state and only makes changes if needed.

The integration pattern has two approaches: Terraform-first and dynamic inventory. In the Terraform-first approach, Terraform provisions the infrastructure and outputs the IP addresses, hostnames, and connection details. These outputs feed into Ansible's inventory — either through a static inventory file generated by Terraform's local_file resource, or through Ansible's AWS dynamic inventory plugin that discovers EC2 instances by tags. The workflow is: Terraform apply creates servers with specific tags, then Ansible runs with the aws_ec2 dynamic inventory plugin filtering by those tags, configuring each server according to its role. In a banking pipeline, this might look like: Terraform creates three EC2 instances tagged role=settlements-processor, and Ansible's playbook installs Java 17, deploys the settlements application JAR, configures the JVM parameters, sets up the Splunk forwarder, and applies CIS hardening — all based on the role tag.

In production at a bank, the separation becomes clear when you consider change frequency and blast radius. Terraform changes are infrequent and high-impact — modifying a VPC peering connection or upgrading an RDS instance affects the entire platform. These changes go through Terraform Enterprise with Sentinel policies, approval gates, and change windows. Ansible changes are more frequent and targeted — updating an application configuration, rotating a certificate, or patching a package on 50 servers. These changes run through Ansible Automation Platform (AWX/Tower) with job templates, RBAC, and audit logging. Both tools store their code in Git, but they have separate repositories, separate CI/CD pipelines, and separate approval workflows because their risk profiles are fundamentally different.

The biggest gotcha is overlapping responsibility. If Terraform creates a security group and Ansible also tries to manage the same security group rules, they fight each other — Terraform's next apply will revert Ansible's changes, and Ansible's next run will override Terraform's state. Define clear boundaries: Terraform manages everything at the cloud API level (resources you see in the AWS console), and Ansible manages everything at the OS level (things you configure via SSH). Another common mistake is using Terraform's provisioner blocks (remote-exec, local-exec) to do configuration management — provisioners run only on resource creation, not on subsequent applies, they have poor error handling, and they mix infrastructure provisioning with configuration in ways that are hard to debug and maintain. Use Terraform to create, Ansible to configure, and keep them in separate pipelines.

Think of Sentinel as a building code inspector who reviews blueprints before construction begins. The architect (engineer) designs the building (infrastructure), submits the blueprints (Terraform plan) to the inspector (Sentinel), and construction (apply) only proceeds if the blueprints comply with all building codes (policies). Unlike a human inspector who might miss details or be inconsistent, Sentinel checks every single blueprint against every single code — automatically, consistently, and without exception.

Sentinel policies in Terraform Enterprise evaluate the Terraform plan, state, and configuration before an apply is permitted. Each policy is a code file written in the Sentinel language that imports Terraform plan data and applies boolean logic to determine pass or fail. Policies are organized into policy sets, and each set is attached to specific workspaces. For a bank, you might have a 'pci-dss-mandatory' policy set attached to all production workspaces, a 'cost-governance' set attached to all workspaces, and a 'network-segmentation' set attached to networking workspaces. Policy sets reference a Git repository, so policies are version-controlled, peer-reviewed, and have their own CI/CD lifecycle.

PCI-DSS compliance policies are the cornerstone for banking. These include: all storage resources (S3, EBS, RDS, EFS) must have encryption at rest enabled using customer-managed KMS keys (not AWS-managed keys), no security group rules can allow ingress from 0.0.0.0/0 on any port, all RDS instances must have automated backups with a minimum 7-day retention, all CloudTrail logs must be enabled and sent to a centralized logging account, all EC2 instances must be deployed in private subnets (no public IP assignment), and all load balancers must have access logging enabled. Each of these requirements becomes a Sentinel policy that evaluates the Terraform plan and blocks the apply if any resource violates the rule.

Sentinel enforcement levels create a tiered governance model. Hard-mandatory policies cannot be overridden by anyone — they represent non-negotiable regulatory requirements like encryption at rest. Soft-mandatory policies can be overridden by workspace administrators with a documented reason — useful for temporary exceptions during migrations or emergency fixes. Advisory policies log warnings but do not block applies — useful for best practices that are not yet mandatory, like tagging standards for new cost centers. In banking, most PCI-DSS policies are hard-mandatory, cost governance policies are soft-mandatory (to allow approved exceptions for high-performance workloads), and new policy rollouts start as advisory for a sprint to identify false positives before becoming mandatory.

Writing effective Sentinel policies requires understanding the Terraform plan structure. The tfplan/v2 import provides access to all resource changes (creates, updates, deletes), their before and after values, and the provider configuration. Policies filter resources by type (aws_s3_bucket, aws_db_instance), then assert conditions on their attributes. For complex policies, you can use Sentinel modules — reusable functions shared across policies. For example, a 'require_tags' module that checks for mandatory tags (cost-center, owner, data-classification, compliance-scope) can be imported by multiple policies. Testing Sentinel policies uses the sentinel test command with mock Terraform plan data, ensuring policies behave correctly before deployment.

The biggest gotcha is policy sprawl and maintenance burden. As the organization adds more policies, engineers face longer feedback cycles — waiting for 50 policies to evaluate on every plan. Keep policies focused and fast by using resource-type filters early in the policy to skip irrelevant resources. Another gotcha is false positives — a policy that blocks legitimate infrastructure changes erodes trust in the governance system and leads to engineers finding workarounds. Invest in thorough policy testing with both passing and failing plan fixtures. Test edge cases like resource updates (not just creates), data sources, and resources managed by different providers. Finally, communicate policy changes clearly — when a new hard-mandatory policy is introduced, give teams a sprint to remediate existing infrastructure before enforcement begins.

Think of drift detection like a nightly security guard doing rounds. The guard has a checklist of how every door, window, and safe should look. If something has changed — a window left open, a safe combination altered, a new lock installed — the guard reports it. Drift detection in Terraform works the same way: scheduled plan runs compare what actually exists in AWS against what Terraform expects, and any discrepancy is flagged for investigation.

Infrastructure drift occurs when the actual state of cloud resources diverges from the Terraform-declared state. This happens through manual console changes (someone modifies a security group via the AWS console), changes by other tools (an automation script modifies a resource that Terraform also manages), auto-scaling events that modify resource attributes, and AWS service updates that change default behaviors. In a banking environment, drift is a compliance risk — if your Terraform code declares that an RDS instance has encryption enabled but someone disables it through the console, your compliance posture is degraded and your Terraform state does not reflect reality.

Terraform Enterprise enables scheduled plan-only runs on workspaces. You configure a workspace to run terraform plan automatically at a set interval — typically nightly for production workspaces and weekly for non-production. The plan compares the current Terraform configuration and state against the actual infrastructure via provider API calls. If the plan detects changes (resources to update, create, or destroy), it means drift has occurred. TFE marks the run as 'planned and finished' with a non-empty plan, and you can configure webhook notifications to alert your team via Slack, PagerDuty, or a custom drift-tracking system.

At enterprise scale, not all drift is equal. A changed tag is low-severity drift that might be auto-remediated. A modified security group rule is high-severity drift that requires immediate investigation — someone may have opened a port that violates PCI-DSS. A deleted resource is critical drift that needs urgent attention. Build a drift classification system: the webhook from TFE sends the plan summary to a Lambda function or custom service that parses the plan output, categorizes each change by resource type and attribute, assigns a severity level, and routes the notification appropriately. Low-severity drift creates a Jira ticket for the next sprint. High-severity drift pages the security team. Critical drift triggers an incident response.

For drift remediation, there are two approaches. Auto-remediation configures TFE to automatically apply the plan when drift is detected, restoring infrastructure to the declared state. This is appropriate for low-risk drifts like tag changes or description updates, but dangerous for high-risk resources — auto-applying a plan that wants to recreate an RDS instance would cause downtime. Selective auto-remediation uses Sentinel policies to evaluate the drift plan: if the only changes are to tags and descriptions, auto-apply; if the plan includes any destroy or replace actions, block and alert. Manual remediation requires a human to review the drift, determine whether the Terraform code or the infrastructure should be updated, and either apply the plan or update the code to match the new reality.

The biggest gotcha is drift detection generating noise that teams ignore. If your scheduled plans consistently show drift from resources that Terraform partially manages (like ASG instance counts that change with auto-scaling), the team learns to dismiss all drift alerts. Use lifecycle ignore_changes blocks in Terraform for attributes that are expected to drift (like ASG desired_count), and ensure your scheduled plans only flag genuine unauthorized changes. Another gotcha is the API rate limiting — running terraform plan across 200 workspaces simultaneously hammers the AWS API. Stagger your scheduled plans across the night, and use workspace tags to group and schedule them in batches. Finally, drift detection only catches drift in resources Terraform manages — resources created manually outside of Terraform are invisible. Complement TFE drift detection with AWS Config rules that detect unmanaged resources.

Think of a private module registry like an internal app store for a bank. Instead of every team building their own RDS setup from scratch (with varying levels of security hardening), the platform team publishes a vetted 'RDS Module' to the internal store. Application teams install it, configure their specific parameters (database name, size), and get encryption, backup, monitoring, and compliance built in. The platform team updates the module when security requirements change, and consumer teams upgrade on their own schedule — just like updating an app on your phone.

A private Terraform module registry serves as the single source of truth for approved infrastructure patterns. In Terraform Enterprise, the private registry is built in — you publish modules from Git repositories with the naming convention terraform-<provider>-<name> (like terraform-aws-eks-cluster or terraform-aws-rds-postgresql). In organizations using open-source Terraform, alternatives include JFrog Artifactory's Terraform provider, a self-hosted registry using the Terraform Registry Protocol, or even Git-based module references with version tags. The key principle is that production infrastructure should only use modules from the private registry, never ad-hoc inline resources — this is enforced via Sentinel policies that check the module source in every Terraform plan.

Semantic versioning is critical for module governance. Every module follows semver (major.minor.patch): patch versions fix bugs without changing behavior, minor versions add features backward-compatibly, and major versions include breaking changes that require consumer updates. When the platform team updates the RDS module to require a new mandatory tag (minor version bump), consumer teams see the new version in the registry but continue using their pinned version until they are ready to upgrade. When a major version changes the module interface (removing an input variable or changing an output format), consumer teams must explicitly update their code. Version constraints in consumer code (version = '~> 2.0' means any 2.x version) allow automatic adoption of patches and minor updates while protecting against breaking changes.

Team ownership is formalized through module CODEOWNERS files and documentation. Each module has an owning team listed in the repository's CODEOWNERS file, ensuring that any PR to the module requires review from the owners. The owning team is responsible for security patching (updating provider versions, fixing CVEs), documentation (README with usage examples, input/output descriptions, and architecture diagrams), testing (automated tests using Terratest or terraform-compliance that run in CI), and deprecation communication (announcing when older versions will lose support). In a banking organization, module ownership maps to infrastructure domains: the networking team owns the VPC and transit gateway modules, the database team owns the RDS and ElastiCache modules, and the platform team owns the EKS and observability modules.

The module development lifecycle follows a structured process. A team identifies a repeated infrastructure pattern (every team needs an S3 bucket with encryption, versioning, access logging, and lifecycle policies). They build a module, test it with Terratest (creating real infrastructure in a sandbox account, validating it, then destroying it), write documentation, and publish version 1.0.0 to the private registry. Consumer teams adopt the module with a pinned version constraint. When a security requirement changes (for example, a new PCI-DSS control requires S3 Object Lock), the module team releases version 1.1.0 with the new feature as an optional input, and version 2.0.0 if the feature must be mandatory (breaking change for consumers not passing the new input). The module team announces the update through internal channels and provides migration guides for major version bumps.

The biggest gotcha is creating modules that are either too opinionated or too flexible. A module that hardcodes the instance type, subnet, and tags is useless because every consumer has different requirements. A module that exposes every single AWS resource argument as an input variable is just a wrapper around the provider with no added value. Good modules encode organizational opinions (encryption is always on, backups are always enabled, monitoring is always configured) while exposing legitimate customization points (instance size, database name, backup retention period). Another gotcha is orphaned modules — modules published to the registry but never updated, with no clear owner. Implement a quarterly module health review where each module is checked for dependency updates, provider compatibility, and active ownership. Deprecate modules that are no longer maintained with clear migration paths to replacements.

Think of Terraform state as a shared bank ledger. If multiple tellers write to the same page at the same time, the numbers get corrupted. If someone photocopies the ledger and takes it home, sensitive account details are exposed. Managing state securely means controlling who can read it, who can write to it, and ensuring only one person writes at a time.

Terraform state contains the full mapping between your HCL code and real infrastructure — including resource IDs, IP addresses, and sometimes sensitive values like database passwords and private keys. In a multi-team environment, this state must be stored remotely (never in Git), encrypted, locked during operations, and access-controlled. The standard pattern on AWS is an S3 bucket with server-side encryption (KMS), versioning enabled for recovery, and a DynamoDB table for state locking.

The locking mechanism prevents concurrent writes. When terraform plan or apply runs, it acquires a lock in DynamoDB using the state file path as the key. If another team member or pipeline tries to run simultaneously, they get a lock error and must wait. This prevents state corruption from concurrent modifications. The lock is released automatically when the operation completes, or can be force-unlocked with terraform force-unlock if a pipeline crashes mid-operation.

At multi-team scale, state should be split by ownership boundary. Each team or component gets its own state file — the networking team manages vpc/terraform.tfstate, the platform team manages eks/terraform.tfstate, and application teams manage their own app/terraform.tfstate. This means a terraform destroy by the app team cannot touch the VPC or EKS cluster. Cross-team references use terraform_remote_state data sources or SSM parameters rather than managing resources across state boundaries.

The non-obvious gotcha is that terraform_remote_state requires read access to the other team's state file, which contains all their sensitive outputs. A safer alternative is writing shared values to SSM Parameter Store or Vault, so consumers get only the specific values they need without accessing the full state. Also, state files should never be committed to Git — even encrypted state contains sensitive metadata that can leak through Git history even after deletion.

Think of assigned seating at a dinner. count is like numbering seats 1 through 10 — if guest #3 cancels, everyone from seat 4 onward shifts down, causing chaos with place cards and meal orders. for_each is like name tags on seats — removing 'Alice' only affects Alice's seat, and everyone else stays put. The difference seems minor until you have production infrastructure attached to those seats.

count creates resources indexed by number: aws_subnet.private[0], aws_subnet.private[1], aws_subnet.private[2]. If you remove the second item from your list, what was index [2] becomes [1], and Terraform sees this as 'destroy [2] and modify [1]' — which means destroying and recreating a subnet that might have running pods on it. This is catastrophic for stateful resources.

for_each creates resources indexed by string key: aws_subnet.private["us-east-1a"], aws_subnet.private["us-east-1b"], aws_subnet.private["us-east-1c"]. If you remove us-east-1b, Terraform only destroys that one subnet. The others are untouched because their keys did not change. This makes for_each the safe default for any resource that has a natural identifier.

Use count only for truly identical resources where the number matters but individual identity does not — like creating N replicas of a test instance that can be destroyed and recreated freely. Use for_each for everything else: subnets (keyed by AZ), IAM users (keyed by name), security group rules (keyed by port or CIDR), DNS records (keyed by hostname), and S3 buckets (keyed by purpose). Converting from count to for_each on existing resources requires terraform state mv to remap indices to keys, or use moved blocks.

The non-obvious gotcha is that for_each requires a set or map — not a list. If you pass a list, you must convert it with toset(). Also, the for_each key becomes part of the resource address, so changing a key destroys and recreates that resource. Choose stable, meaningful keys that will not change over the resource's lifetime. Using the AZ name as a key for subnets is stable. Using an auto-generated ID is not.

Think of reorganizing books on a shelf. You are not buying new books or throwing old ones away — you are just moving them from one shelf to another and updating the library catalog. The physical books (cloud resources) stay exactly the same; only the catalog entry (state file address) changes. If you do not update the catalog, the librarian thinks the book is missing from the old shelf and orders a new one while discarding the one on the new shelf.

When you refactor Terraform code to extract resources into modules, the resource address changes. A resource that was aws_security_group.payments_sg becomes module.networking.aws_security_group.payments_sg. Without telling Terraform about this move, it sees the old address as 'resource to destroy' and the new address as 'resource to create' — which means production downtime while it deletes and recreates your security group.

The moved block, introduced in Terraform 1.1, solves this declaratively. You add a moved block to your configuration that tells Terraform: 'the resource that used to be at address X is now at address Y.' When you run terraform plan, Terraform sees the move instruction, updates the state file to reflect the new address, and shows no create or destroy actions. The moved block can stay in your code permanently as documentation, or be removed after all teams have applied the change.

At production scale, refactoring into modules is a multi-step process. First, write the module and add moved blocks for every resource being relocated. Second, run terraform plan to verify zero creates and zero destroys — only state moves. Third, apply to update the state. Fourth, optionally remove the moved blocks after all environments have applied. For complex refactors involving dozens of resources, use terraform state mv as a batch operation with a script that generates all the state mv commands.

The non-obvious gotcha is that moved blocks only work within the same state file. If you are splitting resources into a completely separate state file (different backend), you need terraform state mv -state-out to export resources, then terraform import in the new state. This two-step process has a window where the resource exists in neither state, so plan carefully and do it during a maintenance window.

Think of a restaurant menu board. The manager updates the board (Terraform code), but a waiter changes the daily special price by hand on the physical sign (console change). The menu board and the sign now disagree. Drift detection is the manager checking the sign against the board every morning, and remediation is deciding whether to update the board or fix the sign.

Infrastructure drift occurs when real-world resources diverge from Terraform's state file. This happens when someone makes changes through the AWS console, another automation tool modifies resources, or AWS auto-applies changes like security patches. Terraform does not continuously monitor infrastructure — it only detects drift when you run terraform plan, which triggers a refresh that reads current resource attributes from cloud APIs.

terraform plan -refresh-only is the detection tool. It reads the current state of every resource in the state file and shows what has changed without proposing any modifications. The output shows 'Objects have changed outside of Terraform' with specific attribute differences. For automated detection, run this command on a schedule (daily via CI) and alert when the exit code indicates drift (terraform plan -refresh-only -detailed-exitcode returns exit code 2 when drift exists). AWS Config rules provide complementary detection for specific compliance violations like open security groups.

Remediation depends on intent. If the manual change was a valid emergency fix (someone tightened a security group during an incident), update the HCL code to include the change and apply. If the change was unauthorized or accidental, run terraform apply with the existing HCL to revert the resource to its declared state. For resources manually created outside Terraform, use terraform import to bring them under management, then write the corresponding HCL.

The non-obvious gotcha is that running terraform apply -refresh-only updates the state file to match reality, which makes Terraform think the manual change is now the desired state. If you later modify the HCL and apply, the manual change persists because it is now the state baseline. Always treat refresh-only as a detection command, not a fix. Also, some drift is expected and harmless — AWS tags added by auto-scaling, EKS-managed security group rules, or Lambda layer versions updated by deployment pipelines. Use lifecycle ignore_changes for attributes managed by external systems.

Think of a building permit process. Before construction (terraform apply) begins, inspectors (policy tools) review the blueprints (terraform plan) for code violations, fire safety issues, and zoning compliance. No permit means no construction. The inspectors do not do the building — they only verify that the plans meet standards before the builder starts work.

Security and compliance checks in Terraform operate at three layers. Static analysis (pre-plan) scans HCL source code for known bad patterns like unencrypted S3 buckets, public security groups, or missing tags. Tools like Checkov, tfsec, and Trivy scan .tf files directly in CI and fail the pipeline if violations are found. This catches issues before Terraform even generates a plan.

Plan-time policy evaluation (post-plan, pre-apply) is the most powerful layer. terraform plan -out=plan.tfplan generates a plan file, then terraform show -json plan.tfplan converts it to JSON. This JSON contains the exact changes Terraform will make — resources to create, modify, or destroy with all their attributes. Policy tools like OPA/Conftest or Sentinel evaluate this JSON against rules: 'no security group may have ingress from 0.0.0.0/0', 'all RDS instances must have encryption enabled', 'no resource may be destroyed in the prod workspace without approval.' If any policy fails, the pipeline blocks apply.

In Terraform Cloud/Enterprise, Sentinel policies are built-in. Policies are categorized as advisory (warning only), soft-mandatory (can be overridden by authorized users), or hard-mandatory (cannot be overridden). This three-tier approach lets teams enforce critical security rules strictly while allowing flexibility for less critical standards. Cost estimation is also available, alerting when a plan would increase monthly spend beyond a threshold.

The non-obvious gotcha is that static analysis alone misses context-dependent violations. A security group rule allowing 0.0.0.0/0 on port 443 is fine for a public ALB but dangerous for an RDS instance. Plan-time policy evaluation sees the full resource context (what type of resource, what VPC, what tags) and can make smarter decisions. Teams should use both layers: static analysis for quick feedback during development, and plan-time policies for authoritative enforcement before apply. Also, policy tests are code too — version them in Git, review them in PRs, and test them against sample plans.

Recovery Process

1. Stop all Terraform operations — prevent further corruption 2. Identify the corruption: terraform plan will usually show the error. Common: Error: Invalid state file 3. Restore from backup: - S3 backend: Use S3 versioning to restore previous state version - aws s3api list-object-versions --bucket my-tf-state --prefix env/prod/terraform.tfstate - Download the last known good version and upload as current 4. For partial corruption: Use terraform state rm to remove the corrupted resource, then terraform import to re-import it 5. Validate: Run terraform plan and verify the diff matches expected state

Common Causes of Unexpected Destroy/Recreate

1. Provider upgrade: New provider version changes a resource attribute from optional to ForceNew. Example: changing aws_instance.ami forces replacement. 2. State key mismatch: Refactoring module structure (renaming modules, moving resources between modules) causes Terraform to see new resources instead of existing ones 3. ForceNew attributes changed: Some attributes can't be updated in-place (e.g., aws_rds_instance.engine, aws_launch_template.name) 4. Backend migration issue: State file doesn't match the current backend configuration 5. Count/for_each index shift: Changing from count to for_each or reordering count indices

Recovery Procedure

Step 1: Backup Before any state surgery, create a backup. With remote backends, download the state file. With S3 backend, versioning provides automatic backups.

Step 2: Inventory Run terraform state list to see all tracked resources. Run terraform plan to see what Terraform thinks needs to change. Resources showing 'destroy' that shouldn't be destroyed are phantom entries. Resources not in the plan but existing in AWS are orphans.

Step 3: Remove phantom resources Use terraform state rm <resource_address> for each phantom resource. This only removes from state, doesn't touch real infrastructure.

Step 4: Import orphaned resources Use terraform import <resource_address> <cloud_resource_id> to bring untracked resources into state. Since Terraform 1.5+, use import blocks in HCL for repeatable imports.

Step 5: Reconcile attributes Run terraform refresh (or terraform apply -refresh-only) to update state attributes from the real cloud state. Review the changes before approving.

Step 6: Validate Run terraform plan and verify it shows minimal or no changes. Any remaining diff should be reviewed carefully.