Terraform Intermediate Interview Questions

Terraform state is the backbone of how Terraform understands what infrastructure it manages. Think of it like a warehouse inventory ledger: without it, workers would walk into the warehouse every day not knowing what is already on the shelves, what was ordered, or what needs restocking. The state file (terraform.tfstate) is that ledger — it records every resource Terraform has created, its current attributes, metadata about dependencies, and the mapping between your HCL resource blocks and the actual cloud API objects.

Internally, the state file is a JSON document containing a version number, a serial counter that increments on every write, a lineage UUID that uniquely identifies a state chain, and an array of resource objects. Each resource entry stores the provider, the resource type, the resource name, the mode (managed or data), and the full set of attributes returned by the provider API after creation. When you run terraform plan, Terraform reads this state, calls the cloud APIs to refresh the actual status of each resource, and then computes the diff between desired (your HCL) and actual (the refreshed state). Without state, Terraform would have no way to know that aws_rds_instance.payments_db already exists and would try to create a duplicate every time.

Local state works fine for a solo developer experimenting, but it becomes dangerous in production for several reasons. First, if two engineers run terraform apply simultaneously against local state files, they can create conflicting resources or corrupt state entirely — there is no locking mechanism. Second, if your laptop dies or someone accidentally deletes the state file, you lose the mapping between code and infrastructure, making it extremely difficult to recover. Third, local state may contain sensitive outputs like database passwords or API keys stored in plaintext on a developer workstation.

Remote state solves all of these problems. When you configure a backend like S3 with DynamoDB locking, the state file lives in a durable, versioned object store. DynamoDB provides a distributed lock so that only one terraform apply can run at a time, preventing race conditions. S3 versioning gives you automatic backup of every state revision, so you can roll back if something goes wrong. Additionally, remote state enables the terraform_remote_state data source, which lets one Terraform project read outputs from another — for example, a networking project can export VPC IDs that an application project consumes.

In production, teams typically enforce remote state from day one using a backend configuration block, require encryption at rest and in transit, restrict access via IAM policies, and enable state locking. Ignoring remote state is one of the most common causes of Terraform disasters in growing organizations.

A Terraform module is essentially a directory containing .tf files that encapsulate a logical group of resources. Think of modules like functions in programming: they take inputs (variables), do something (create resources), and return outputs. The root module is your working directory where you run terraform commands, and any module you call from there is a child module. When you write module "payments_vpc" { source = "./modules/vpc" }, Terraform loads that directory as an isolated configuration unit with its own namespace.

Internally, when Terraform processes a module call, it creates a separate resource namespace prefixed with module.payments_vpc. Resources inside the module cannot directly access resources outside it — they communicate only through input variables and output values. This enforced encapsulation is what makes modules safe to reuse. Terraform also supports module sources from Git repositories, the Terraform Registry, S3 buckets, and HTTP URLs, enabling organization-wide module libraries.

Good module design starts with the single-responsibility principle. A VPC module should create a VPC, subnets, route tables, and NAT gateways — it should not also create your RDS database. Each module should represent one logical infrastructure component. Variables should have descriptions, type constraints, and sensible defaults where possible. For example, a VPC module might default to three availability zones and a /16 CIDR block but allow overrides. Outputs should expose the identifiers that downstream modules need — VPC ID, subnet IDs, security group IDs — nothing more.

One critical design principle is avoiding hardcoded provider configurations inside modules. The module should inherit the provider from the calling module, not declare its own. This allows the same module to be used across multiple AWS accounts or regions by simply changing the provider in the root module. Similarly, avoid hardcoding backend configurations or environment-specific values like account IDs inside modules.

Version pinning is essential for module stability. When sourcing modules from a registry or Git, always pin to a specific version or Git tag. Using version = "~> 2.0" ensures you get patch updates but not breaking major version changes. Without version pinning, a terraform init on Monday might pull different module code than the same command on Friday, leading to unpredictable infrastructure changes.

Production-grade modules also include validation blocks for input variables, meaningful error messages, comprehensive README documentation, and example configurations. Teams that invest in a well-designed internal module library see dramatic reductions in infrastructure provisioning time and configuration drift across environments.

Understanding the difference between plan and apply is fundamental, but the internals reveal much more than just 'one previews, the other executes.' Think of terraform plan like a restaurant printing a receipt before charging your card — it shows exactly what will happen so you can review it. terraform apply is when the charge actually goes through.

When you run terraform plan, Terraform performs several steps internally. First, it loads the configuration by parsing all .tf files in the current directory and resolving module sources. Second, it reads the current state file to understand what resources already exist. Third, it performs a state refresh — making API calls to every cloud provider to check the actual status of each managed resource and updating the in-memory state with real attributes. This refresh step is crucial because someone might have manually changed a security group rule outside of Terraform. Fourth, Terraform builds a dependency graph of all resources, computes the diff between desired state (your HCL) and actual state (refreshed), and produces an execution plan showing creates, updates, and destroys with specific attribute changes.

The plan output uses a clear notation: + for create, ~ for update in-place, - for destroy, and -/+ for destroy-then-recreate (also called a forced replacement). When you see -/+ next to your production database, that is the moment you should stop and investigate — it means Terraform wants to destroy and recreate that resource, which could mean data loss.

terraform apply by default runs a plan first and asks for confirmation before proceeding. Once confirmed, Terraform walks the dependency graph and makes real API calls to create, update, or destroy resources. It processes independent resources in parallel (up to 10 by default, configurable with -parallelism) and sequential resources in dependency order. After each resource operation completes, Terraform immediately writes the updated state file, ensuring that even if apply is interrupted midway, the state reflects what was actually created.

A critical production practice is using saved plan files. You run terraform plan -out=tfplan to save the plan to a binary file, review it, and then run terraform apply tfplan. This guarantees that what you reviewed is exactly what gets applied — no re-computation, no changes from someone else's commit sneaking in between plan and apply. In CI/CD pipelines, this two-stage approach is essential. The plan stage runs in a pull request for review, and the apply stage runs only after merge using the exact saved plan.

One subtle gotcha: terraform apply without a saved plan will re-compute the plan at apply time, meaning the infrastructure could have changed between when you reviewed the plan output and when apply runs. In fast-moving environments with multiple teams, this gap can cause surprises. Always use saved plans in production workflows.

Terraform workspaces are a built-in mechanism for managing multiple instances of the same infrastructure configuration. Think of workspaces like branches in a photo editing app — you have the same source image but can apply different filters and adjustments to each branch independently. Each workspace gets its own state file, so resources in the 'dev' workspace are completely isolated from resources in the 'prod' workspace, even though they share the same Terraform code.

Internally, workspaces work by modifying the state file path. When using the default local backend, Terraform stores state files in a terraform.tfstate.d/ directory with subdirectories for each workspace. With an S3 backend, workspace state files are stored under the key prefix with the workspace name appended — for example, payments-platform/dev/terraform.tfstate and payments-platform/prod/terraform.tfstate. The terraform.workspace variable is available in your HCL code, letting you conditionally set values based on the active workspace.

Workspaces shine when your environments are structurally identical but differ in scale or configuration. If dev, staging, and production all need the same VPC, RDS cluster, ECS service, and ALB, but dev uses smaller instance types and fewer replicas, workspaces with conditional expressions or workspace-specific tfvars files work beautifully. You can use terraform.workspace in locals to set instance sizes, replica counts, and CIDR ranges per environment.

However, workspaces have significant limitations that push many teams toward separate directories. First, all environments share the same provider configuration — you cannot easily use different AWS accounts per workspace without complex provider alias tricks. Second, if your production environment has additional resources that dev does not need (WAF rules, CloudFront distributions, compliance monitoring), you end up with count = terraform.workspace == "prod" ? 1 : 0 scattered throughout your code, which becomes unreadable. Third, workspaces provide no protection against accidentally running apply in the wrong workspace. A sleep-deprived engineer who forgets to run terraform workspace select prod before applying a production hotfix could accidentally modify the dev environment.

Separate directories (often called the directory-per-environment pattern) give you complete isolation. Each environment has its own directory with its own backend configuration, provider configuration, and state. This means prod can use a different AWS account, different provider version constraints, and completely different resource compositions. The tradeoff is code duplication — you need to keep common modules in sync across directories.

The modern consensus in the Terraform community is to use workspaces for lightweight environment differentiation within a single account and team, and separate directories (or separate Terraform Cloud workspaces with VCS integration) for production-grade multi-account setups. Many teams use a hybrid approach: modules contain the shared logic, and each environment directory calls those modules with environment-specific variables.

Data sources in Terraform are read-only queries against your infrastructure provider's API. Think of the difference like this: a resource block is like writing a check at a bank — it changes your account balance. A data source is like checking your account statement — it only reads information without modifying anything. Both interact with the same bank (cloud provider), but their intentions and side effects are fundamentally different.

Internally, when Terraform encounters a data block, it makes an API call to the provider during the refresh phase of plan or apply and stores the returned attributes in state. The provider plugin handles translating the data source's filter criteria into the appropriate API call — for example, data.aws_ami.ubuntu_latest with filters for name and owner translates into an EC2 DescribeImages API call with those filters. The returned attributes become available for reference in other parts of your configuration.

Data sources serve several critical purposes in production Terraform code. First, they bridge the gap between managed and unmanaged infrastructure. If your networking team manages VPCs outside of your Terraform project, you use data.aws_vpc to look up the VPC by tags or ID rather than hardcoding the VPC ID. This makes your code portable and resilient to infrastructure changes. Second, data sources enable dynamic configuration — instead of hardcoding the latest Ubuntu AMI ID and updating it manually, data.aws_ami queries for the most recent AMI matching your criteria every time you plan. Third, they connect separate Terraform projects through terraform_remote_state data sources.

A subtle but important difference is lifecycle behavior. Resources have full lifecycle management: Terraform creates them, tracks them in state, updates them when configuration changes, and destroys them when removed from code. Data sources have no lifecycle — they are read during plan/apply and their results are cached in state, but Terraform never creates, updates, or destroys the underlying infrastructure. If a data source query returns no results, Terraform fails with an error rather than creating anything.

One common gotcha is circular dependencies between data sources and resources. If you create an AWS security group with a resource block and then try to use a data source to read that same security group in the same configuration, you may hit issues during the first apply because the data source tries to read something that does not exist yet. The rule of thumb is: if Terraform creates it, reference it directly as a resource; if something else creates it, use a data source.

Data sources also respect provider authentication and permissions, so your Terraform IAM role needs read access to whatever the data source queries. In production, teams often need broader read permissions than write permissions because data sources query across account boundaries — for example, reading a shared services account's Route53 hosted zone to create DNS records in it.

Terraform's dependency management is one of its most powerful features, and understanding it deeply is key to writing correct infrastructure code. Think of dependencies like a construction project: you cannot install electrical wiring (resource B) until the walls are framed (resource A). Terraform figures out this ordering automatically by analyzing which resources reference attributes of other resources.

Implicit dependencies are the primary and preferred mechanism. When you write aws_instance.payments_api.subnet_id = aws_subnet.private_payments.id, Terraform analyzes this reference and understands that the subnet must exist before the instance can be created. It adds an edge in the dependency graph from aws_subnet.private_payments to aws_instance.payments_api. During apply, Terraform walks this directed acyclic graph (DAG) in topological order, creating resources with no dependencies first and working its way up. For destroy operations, Terraform reverses the graph — it destroys the instance before the subnet, because the instance depends on the subnet.

Internally, Terraform's graph builder collects all resource and data source blocks, resolves all interpolation references, and constructs a DAG where nodes are resources and edges are dependencies. It then validates the graph is acyclic — if you accidentally create a circular dependency (A references B, B references A), Terraform detects this and fails with an error showing the cycle. The graph also determines parallelism: resources at the same depth in the graph with no dependencies between them can be created simultaneously.

Explicit dependencies using depends_on are necessary when a dependency exists but is not visible through attribute references. The classic example is an IAM role and an EC2 instance that assumes that role. If the instance references the role ARN directly, the implicit dependency works. But if the instance uses an instance profile that was created elsewhere, and the role's permissions are needed at boot time for a user_data script, Terraform might try to launch the instance before the IAM role's policy attachments are complete. AWS IAM has eventual consistency — a policy attachment might take seconds to propagate. Adding depends_on = [aws_iam_role_policy_attachment.payments_api_s3_access] forces Terraform to wait.

However, depends_on should be used sparingly because it has a hidden cost. When you add depends_on to a resource, Terraform treats it as a more opaque dependency — during plan, it may not be able to determine the full set of changes, which can lead to unnecessary replacements or overly conservative planning. It also makes your code harder to understand because the dependency reason is not self-documenting like an attribute reference.

Another important dependency concept is resource ordering during updates. If you change a security group rule that an RDS instance references, Terraform understands that the security group must be updated before or after the RDS instance depending on the nature of the change. Terraform's graph handles this automatically for most cases, but understanding the graph helps you predict apply behavior and debug issues when operations fail mid-apply.

terraform import is the bridge between manually-created infrastructure and Infrastructure as Code. Think of it like adopting a rescue dog — the dog already exists, you just need to register it in your system and start managing it properly. The resource is out there in AWS or GCP, and terraform import tells Terraform 'this resource exists, start tracking it in state.'

Internally, terraform import works by taking a resource address (the Terraform resource block identifier) and a cloud provider resource ID (like an AWS instance ID), then making API calls to fetch the current attributes of that resource and writing them into the state file. Critically, import does NOT generate the HCL configuration for you — it only updates state. This means you must first write the resource block in your .tf files that matches the imported resource, then run the import, and then iterate by running terraform plan to see what attributes are missing or mismatched in your HCL until the plan shows no changes.

The traditional import workflow is tedious but important: write an empty or partial resource block, run terraform import aws_rds_cluster.payments_db payments-db-prod-cluster-id, then run terraform plan repeatedly while adding missing attributes to your HCL until the plan is clean. Each plan iteration will show you attributes that differ between your config and the actual resource — things like backup retention periods, maintenance windows, or parameter groups that you forgot to specify.

Starting with Terraform 1.5, the import block was introduced as a declarative alternative. Instead of running an imperative CLI command, you add an import block to your configuration that specifies the resource address and cloud ID. When you run terraform plan, it shows the import as part of the plan, and terraform apply executes it. This is a significant improvement because imports become part of your version-controlled code, reviewable in pull requests, and repeatable across environments.

Common scenarios where you need terraform import include: migrating a legacy infrastructure to Terraform (the most common case), recovering after accidental state deletion, bringing a resource that was created via the AWS console clickops into Terraform management, and splitting a large Terraform project into smaller ones where some resources need to move to a different state file.

There are important gotchas with import. Not all resource attributes are importable — some providers or resource types do not support import at all, and you will get an error. Some resources require specific ID formats that are not obvious — for example, an AWS security group rule import requires a combination of security group ID, type, protocol, port range, and CIDR. After importing, you must verify with terraform plan that your configuration matches the imported resource, because any drift will be 'corrected' on the next apply, potentially causing outages. Finally, import does not handle dependencies — if you import a subnet but not the VPC it belongs to, your configuration must use a data source for the VPC or import it as well.

Managing sensitive values in Terraform is one of the most critical aspects of production infrastructure and one of the most commonly mishandled. Think of it like handling classified documents in an office — you need secure storage (Vault), controlled access (IAM policies), no leaving copies around (state encryption), and no reading aloud in public (plan output suppression).

The first layer of defense is the sensitive variable flag introduced in Terraform 0.14. When you mark a variable as sensitive = true, Terraform redacts its value from plan and apply output, replacing it with (sensitive value). This prevents accidental exposure in CI/CD logs, terminal output shared in screenshots, or plan files sent for review. However, this is purely a display-level protection — the actual value is still stored in plaintext in the state file. Many engineers mistakenly believe the sensitive flag encrypts the value, but it does not.

The second and more important layer is keeping secrets out of your Terraform code and version control entirely. Never put passwords, API keys, or tokens in .tf files, .tfvars files, or any file committed to Git. Even if you add them to .gitignore, they can end up in Git history, CI/CD caches, or developer workstations. Instead, use external secret management systems.

HashiCorp Vault is the gold standard for dynamic secrets with Terraform. The Vault provider allows Terraform to read secrets from Vault at plan/apply time using data sources. Better yet, Vault can generate dynamic, short-lived database credentials that automatically expire — meaning even if the state file is compromised, the stolen credentials will be invalid within hours. The Vault provider authenticates using methods like AppRole, AWS IAM, or Kubernetes service accounts, avoiding the chicken-and-egg problem of needing a secret to access secrets.

AWS Secrets Manager and SSM Parameter Store are alternatives that integrate natively with AWS. You store secrets there via the AWS console or CLI, and Terraform reads them with data sources. This approach keeps the secret out of your HCL code and .tfvars files but still stores the value in Terraform state after it is read.

The state file itself is the elephant in the room. Regardless of how carefully you manage inputs, Terraform state contains every attribute of every resource, including sensitive outputs like database connection strings, private keys, and IAM access keys. This is why remote state with encryption at rest (S3 server-side encryption, Terraform Cloud encryption) and strict access controls is non-negotiable. In production, teams typically restrict state file access to CI/CD service accounts and a small group of senior engineers.

Environment variables (TF_VAR_name) provide another mechanism for passing secrets without files. CI/CD systems like GitHub Actions, GitLab CI, and Jenkins can inject secrets as environment variables that Terraform reads automatically. This avoids writing secrets to disk entirely, though they may appear in process listings or CI/CD audit logs.

For the most sensitive operations, teams combine multiple strategies: Vault generates dynamic credentials, Terraform reads them via data sources, the sensitive flag suppresses output, state is encrypted and access-controlled, and CI/CD pipelines run in ephemeral containers that are destroyed after each run.

Both count and for_each are Terraform meta-arguments for creating multiple instances of a resource from a single block, but they differ fundamentally in how they identify and track instances. Understanding this difference prevents one of the most common Terraform disasters in production.

Think of count like numbered seats in a movie theater — seat 0, seat 1, seat 2. If the person in seat 1 leaves and everyone shifts down, the person who was in seat 2 is now in seat 1, and Terraform thinks they are a different person. Think of for_each like named reserved seats — 'Alice's seat', 'Bob's seat', 'Carol's seat'. If Bob leaves, Alice and Carol keep their seats unchanged.

With count, resources are identified by their numeric index: aws_iam_user.payments_engineers[0], aws_iam_user.payments_engineers[1], etc. When you define count = length(var.engineer_names) and the list is ["alice", "bob", "carol"], index 0 is Alice, 1 is Bob, 2 is Carol. If you remove Bob from the middle of the list, the new list becomes ["alice", "carol"]. Now index 1 maps to Carol instead of Bob. Terraform sees that index 1 changed from Bob to Carol and index 2 disappeared, so it plans to update index 1 (changing Bob's user to Carol's) and destroy index 2 (Carol's original user). In production, this means Carol's IAM user gets destroyed and recreated, losing all her attached policies, access keys, and MFA configuration.

With for_each, resources are identified by string keys: aws_iam_user.payments_engineers["alice"], aws_iam_user.payments_engineers["bob"], aws_iam_user.payments_engineers["carol"]. When you remove Bob, Terraform only destroys aws_iam_user.payments_engineers["bob"] — Alice and Carol are untouched because their keys have not changed. This is dramatically safer and more predictable.

for_each accepts either a map or a set of strings. When you pass a map, each.key gives you the map key and each.value gives you the map value, allowing you to associate configuration data with each instance. When you pass a set, each.key and each.value are both the set element. You cannot pass a list directly to for_each — you must convert it to a set with toset() because for_each requires unique keys.

There are still valid use cases for count. The most common is the conditional resource pattern: count = var.enable_monitoring ? 1 : 0 creates the resource only when the variable is true. This is clean, readable, and does not suffer from the index-shift problem because there is only one instance. count is also appropriate when creating a fixed number of identical resources where order does not matter and the count will never change — for example, three NAT gateways, one per availability zone, created in a predictable order.

A subtle gotcha with for_each is that the keys must be known at plan time. You cannot use for_each with a map whose keys come from a resource that has not been created yet — Terraform needs to know all the instance keys to build the dependency graph. If you hit this limitation, you may need to restructure your code or use a targeted apply to create the dependency first.

In modern Terraform, the best practice is to default to for_each for collections and reserve count for boolean conditionals. This prevents the index-shift problem and makes your code more self-documenting since each instance has a meaningful name instead of a number.

Terraform's provider plugin system is the architectural foundation that makes Terraform extensible to any infrastructure platform. Think of Terraform core as a universal remote control — it understands the concept of channels, volume, and power, but it needs a specific adapter (provider plugin) for each TV brand. The AWS provider is the adapter for AWS, the Azure provider for Azure, and the Kubernetes provider for Kubernetes. Without providers, Terraform core cannot manage any infrastructure.

Internally, providers are standalone Go binaries that communicate with Terraform core via gRPC (Google Remote Procedure Call) over a local socket. When you run terraform init, Terraform reads the required_providers block, downloads the specified provider binaries from the configured registry (default: registry.terraform.io), verifies their SHA256 checksums against signed hash files, and stores them in the .terraform/providers directory. The provider binary is platform-specific — there are separate builds for linux_amd64, darwin_arm64, and other OS/architecture combinations.

When terraform plan or apply runs, Terraform core starts each required provider as a child process. The provider plugin registers its resource types and data source types, along with their schemas — essentially telling Terraform core 'I can manage these resource types, and here are the attributes each one supports.' Terraform core then calls provider functions over gRPC for specific operations: ValidateResourceConfig (check if the HCL is valid), PlanResourceChange (compute what would change), ApplyResourceChange (make the actual API call), ReadResource (refresh current state), and ImportResourceState (handle imports).

Provider version constraints are crucial for stability. The required_providers block lets you pin versions using constraint syntax: version = "~> 5.0" means any 5.x version but not 6.0. Without version constraints, terraform init downloads the latest version, which might include breaking changes. The .terraform.lock.hcl file (dependency lock file) records the exact versions and checksums of providers used, ensuring consistent installations across team members and CI/CD environments. This file should be committed to version control.

Provider configuration blocks handle authentication and regional settings. The AWS provider, for example, accepts region, profile, assume_role, and other authentication parameters. You can configure multiple instances of the same provider using aliases — for example, one AWS provider for us-east-1 and another aliased provider for eu-west-1. Resources can then specify which provider alias to use with the provider meta-argument.

The provider ecosystem is vast. HashiCorp maintains official providers for major platforms (AWS, Azure, GCP, Kubernetes). Partner providers are maintained by third-party companies but listed on the registry. Community providers are maintained by individuals. You can also write custom providers using the Terraform Plugin SDK or the newer Terraform Plugin Framework, which is useful for internal platforms or APIs without public providers.

One production consideration is provider caching and air-gapped environments. In restricted networks, you cannot download providers from the public registry. Terraform supports filesystem mirrors, network mirrors, and the terraform providers mirror command to pre-download providers for offline use. Many enterprises run an internal Artifactory or Nexus instance as a provider mirror.

Using tfvars files for multi-environment management is like having a single restaurant menu (Terraform code) with portion size options (tfvars): the recipe stays the same whether you order a small (dev), medium (qa), large (uat), or extra-large (prod) — only the quantities change. This pattern eliminates code duplication while giving each environment its own tunable parameters.

The variable definition layer starts in variables.tf, where you declare every configurable parameter with a type constraint, description, and optionally a default value. Variables without defaults are mandatory and must be provided by the tfvars file. Use object types for related parameters: instead of separate variables for each RDS setting, create a variable of type object({ instance_class = string, allocated_storage = number, multi_az = bool, backup_retention_days = number }) called database_config. This groups related values and makes tfvars files self-documenting.

The tfvars files live in an environments/ directory: environments/dev.tfvars, environments/qa.tfvars, environments/uat.tfvars, environments/prod.tfvars. Each file sets the same variables with environment-appropriate values. Dev might use db.t3.medium with 20GB storage and single-AZ, while Prod uses db.r6g.2xlarge with 500GB and multi-AZ enabled. The key principle is that tfvars files contain only values, never logic. Conditional logic belongs in the Terraform code itself using ternary expressions or lookup maps.

The CI/CD pipeline selects the correct tfvars file based on the target environment. A typical command is terraform plan -var-file=environments/prod.tfvars -out=prod.tfplan. The pipeline determines the environment from the Git branch (feature/* uses dev.tfvars, release/* uses uat.tfvars, main uses prod.tfvars) or from a pipeline variable. This pattern integrates cleanly with the approval workflow: the plan output shows exactly what will change in the target environment before anyone approves.

For complex configurations, layer multiple tfvars files. Use a common.tfvars for values shared across all environments (organization name, DNS domain, tagging standards), then overlay environment-specific files: terraform plan -var-file=common.tfvars -var-file=environments/prod.tfvars. Later files override values from earlier ones, so environment-specific settings take precedence over common defaults.

The critical gotcha is tfvars file drift. Over time, someone adds a new variable to dev.tfvars but forgets to add it to prod.tfvars. Without a default value, the prod plan fails. Prevent this by treating tfvars files as a matrix: every variable declared without a default must appear in every tfvars file. Enforce this with a CI check that parses variables.tf for required variables and validates their presence in all tfvars files. Another gotcha is accidentally committing sensitive values (database passwords, API keys) in tfvars files. Sensitive values should come from environment variables (TF_VAR_*), HashiCorp Vault, or AWS Secrets Manager data sources — never from tfvars files in version control.

Choosing between workspaces and directory-based separation is like choosing between a multi-tenant apartment building (workspaces) and separate houses (directories). The apartment building shares plumbing and electrical systems — cheaper and easier to maintain, but a pipe burst affects everyone. Separate houses cost more to build and maintain, but a problem in one house never reaches another.

Terraform workspaces create named instances of state within the same backend configuration. When you run terraform workspace new qa, Terraform creates a new state file at env:/qa/terraform.tfstate in your backend (S3 key prefix changes to include the workspace name). The configuration stays identical — same main.tf, same modules — and you use terraform.workspace in conditionals or lookups to vary behavior. For example, locals { instance_type = terraform.workspace == "prod" ? "m5.2xlarge" : "t3.large" }. The advantage is zero code duplication: a module upgrade applies to all environments by running apply in each workspace sequentially.

Directory-based separation creates independent root configurations per environment: infrastructure/envs/dev/main.tf, infrastructure/envs/prod/main.tf. Each directory has its own backend configuration, its own provider setup, and its own terraform.tfvars. Shared logic lives in modules referenced by relative path or registry source. The directories might look identical initially, but they can diverge intentionally — Prod might have a WAF module that Dev does not, or Dev might test a newer provider version before promoting it.

The workspace approach has several tradeoffs. On the positive side: single source of truth for infrastructure code, atomic module upgrades, and less repository sprawl. On the negative side: a syntax error in main.tf breaks all environments simultaneously, there is no way to run different Terraform or provider versions per workspace, and terraform.workspace conditionals scattered through the code create hidden complexity. Most critically, a careless terraform apply in the wrong workspace can destroy production — there is no structural guardrail preventing this, only the workspace prompt.

Directory-based separation trades DRY for safety. Each environment is fully independent: you can upgrade Dev to Terraform 1.8 while Prod stays on 1.7, you can test a new module version in QA without touching UAT, and a broken configuration in one directory cannot affect another. The cost is synchronization: when you fix a bug in the Dev VPC module call, you must remember to propagate the fix to QA, UAT, and Prod directories. Tooling like Terragrunt mitigates this by generating directory-based configurations from a DRY template, giving you the isolation of directories with the maintainability of workspaces.

The production recommendation for most teams at Value Momentum's scale is a hybrid approach: use directory-based separation for major infrastructure boundaries (networking, EKS cluster, databases) and workspaces within each directory for environment selection only when the configurations are truly identical. Never use workspaces as a substitute for separate AWS accounts — they operate within a single provider configuration and do not provide IAM or network isolation.

Choosing between a monorepo and multi-repo for Terraform is like choosing between a single warehouse with organized aisles (monorepo) versus separate warehouses per product line (multi-repo). The single warehouse is easier to navigate and keeps inventory consistent, but a forklift accident in one aisle can block the whole building. Separate warehouses provide isolation but make it harder to share parts and coordinate shipments.

The monorepo approach stores all Terraform code in a single repository: shared modules in modules/, environment configurations in envs/dev/, envs/qa/, envs/prod/, and CI/CD pipeline definitions alongside the code. The primary advantage is atomic changes: when you update a shared module and its callers, everything is in one PR, reviewable together. Cross-cutting concerns like provider version upgrades, backend configuration changes, or tagging policy updates are a single commit. Module development and consumption happen in the same repository, eliminating the version publishing ceremony.

The multi-repo approach creates separate repositories per environment or per infrastructure domain: infra-networking, infra-eks, infra-databases. Each repository has its own CI/CD pipeline, its own access controls, and its own release cadence. The advantage is strict blast radius isolation: a broken CI pipeline in the networking repo cannot block database deployments. Access control is more granular — you can give database administrators write access to infra-databases without exposing networking code.

For most teams at the scale of 4 environments (Dev, QA, UAT, Prod), the monorepo with environment directories is the pragmatic choice. The branching strategy that works best is trunk-based development with short-lived feature branches. Main (or trunk) is always deployable — it represents the current desired state of all environments. Engineers create feature branches from main, make changes in the relevant environment directory, open a PR, and the CI pipeline runs terraform plan against the affected environments.

The promotion flow works through directory-level changes, not branch-based promotion. An engineer develops a new VPC peering configuration in envs/dev/peering.tf, merges to main, and the dev pipeline applies it. After validation in dev, they create a new PR that adds the same configuration to envs/qa/peering.tf (possibly with different tfvars). This continues through UAT and Prod. Each environment change is a separate PR with its own review and approval cycle.

The critical branching anti-pattern to avoid is environment branches (dev branch, qa branch, prod branch) where you promote by merging dev into qa into prod. This creates merge conflicts, diverging code paths, and the nightmare scenario where a hotfix to prod must be cherry-picked back through all environment branches. Trunk-based development with directory separation eliminates this entirely.

For the multi-repo approach, use Git tags or releases to version shared modules. Consumer repositories reference modules via Git source URLs with pinned tags: source = "git::https://github.com/org/tf-modules.git//vpc?ref=v2.2.0". Promotion happens by bumping the version tag in each environment's module source, creating a clear audit trail of what version each environment runs.

Naming conventions in Terraform are like street addresses in a city: without a consistent scheme, you end up with two buildings called 'Main Office' and no one knows which is production and which is staging until something breaks. A well-designed naming convention makes every resource self-describing — you should be able to look at a resource name in a CloudWatch alarm or a billing report and immediately know its environment, project, component, and owner.

The naming pattern should follow a hierarchical structure: {project}-{component}-{environment}-{region-short}. For example: payments-api-alb-prod-use1 (project=payments, component=api-alb, environment=prod, region=us-east-1 abbreviated). This pattern works across most AWS resource types, though some have character limits (S3 bucket names max at 63 characters, IAM role names at 64). Build a naming module that accepts these components and outputs formatted names, handling truncation and character restrictions per resource type.

The naming module centralizes convention enforcement. Every resource in your Terraform code calls this module to generate its name rather than constructing names inline. This ensures consistency: if the team decides to change the naming format (adding a cost center code, for example), you update one module and every resource follows suit. The module also handles AWS-specific constraints: S3 buckets cannot have underscores, security group names cannot start with sg-, and some resources require globally unique names while others only need account-level uniqueness.

Environment embedding is the critical collision-prevention mechanism. When Dev and QA share an AWS account (common in early-stage projects), every resource must include the environment in its name. Without this, a terraform apply in dev that creates a security group called payments-api-sg will conflict with the same apply in qa trying to create the same name. The conflict can manifest as Terraform errors (resource already exists) or worse — Terraform adopting the existing resource into its state, giving one environment control over another's security group.

Tag standardization complements naming. While names are visible in the console and logs, tags power cost allocation, automation, and compliance. Define a standard tag set: Environment, Project, Component, ManagedBy (always 'terraform'), Team, CostCenter. Use the AWS provider's default_tags block to automatically apply these to every resource without repeating them in each resource block. This guarantees no resource escapes tagging, even if an engineer forgets to add tags to a new resource.

The gotcha most teams hit is naming convention drift over time. Early resources use one pattern, new resources use an updated pattern, and the codebase becomes inconsistent. Prevent this by adding tflint rules or OPA policies that validate resource names against a regex pattern. Run these checks in CI so non-conforming names fail the pipeline before reaching any environment. Another gotcha is renaming existing resources: changing a resource name in Terraform causes a destroy-and-recreate cycle unless you use a moved block or terraform state mv to update the state mapping.

Think of open-source Terraform as a skilled carpenter with excellent tools who works alone from blueprints. Terraform Enterprise is a construction firm — it has the same carpenter, but adds project managers (RBAC), building inspectors (Sentinel policies), a parts catalog (private registry), apprentice supervision (workspace permissions), and a complete paper trail of every nail hammered (audit logs). A solo developer building a shed does not need a construction firm. A bank building a skyscraper absolutely does.

Open-source Terraform provides the core functionality: HCL language for defining infrastructure, a provider ecosystem for interacting with cloud APIs, state management, plan and apply workflows, and module reuse. When a single engineer or small team manages infrastructure, open-source Terraform with a remote state backend (S3 + DynamoDB) works well. The limitations emerge at scale: who can run terraform apply against production? How do you enforce that all S3 buckets have encryption enabled? How do you share vetted modules across 20 teams without everyone copy-pasting and diverging? How do you prove to auditors that every infrastructure change was reviewed, approved, and logged? Open-source Terraform has no answers to these questions — it is a tool, not a platform.

Terraform Enterprise addresses these gaps through several key features. Workspaces provide isolated environments with their own state, variables, and permissions — the payments-api infrastructure can be in one workspace with access limited to the payments team, while the settlements-db workspace is restricted to the database team. Sentinel policies run before every apply and enforce organizational rules as code — 'all RDS instances must have encryption at rest,' 'no IAM policies can use wildcard actions,' 'all resources must have cost-center and owner tags.' These policies are version-controlled, peer-reviewed, and provide automated compliance enforcement that auditors love. The private module registry lets the platform team publish vetted, hardened modules (like an approved EKS cluster configuration) that application teams consume — ensuring consistency without restricting autonomy.

Remote execution in TFE solves the 'works on my machine' problem. Instead of engineers running terraform apply from their laptops (with their personal AWS credentials and whatever Terraform version they have installed), all plans and applies execute in TFE's managed runners with consistent Terraform versions, standardized provider credentials (injected via workspace variables), and no local state. This eliminates credential sprawl — engineers never need direct AWS access for infrastructure changes, because TFE holds the credentials and applies changes on their behalf. For banking, this is a massive security improvement: credentials are centralized, rotated, and never touch developer machines.

In production at a bank, TFE becomes the control plane for all infrastructure changes. The typical workflow is: engineer creates a branch, makes infrastructure changes, opens a PR. TFE runs a speculative plan on the PR (visible as a GitHub check), showing exactly what will change. Team members review the plan diff alongside the code diff. After PR approval and merge, TFE runs the real plan and waits for workspace-level approval (configurable — some workspaces auto-apply, production workspaces require manual confirmation from an authorized approver). Sentinel policies gate the apply, and the entire execution is logged with timestamps, user identity, plan output, and apply results. These logs are retained for compliance and can be exported to SIEM systems for security monitoring.

The gotcha is cost and operational overhead. TFE is expensive — it requires either a self-hosted installation (on-premise or in your cloud account) or a Terraform Cloud Business subscription. Self-hosted TFE needs its own infrastructure (compute, database, object storage), monitoring, backup, and upgrades. Many organizations start with Terraform Cloud (the SaaS version) for its lower operational burden and migrate to self-hosted TFE only when data residency requirements or network isolation mandates require it. Another common mistake is over-governing — applying strict Sentinel policies and manual approval gates to every workspace, including development and sandbox environments, which slows down experimentation. Use tiered governance: strict policies and approvals for production, advisory-only policies for staging, and minimal controls for development.

Think of Ansible Automation Platform as a universal remote control for every server in your bank — whether it is a Red Hat Linux box in AWS, a Windows Server in your on-premise VMware cluster, or a SUSE machine in Azure. Without it, every team has their own scripts, their own credentials, and their own way of setting up servers. With AAP, there is one platform that speaks every server's language, follows the same playbooks, and logs every action for compliance.

Ansible Automation Platform (formerly Ansible Tower/AWX) provides the enterprise features that raw Ansible lacks: a web UI and API for managing automation, RBAC for controlling who can run what against which servers, credential management that securely stores SSH keys, WinRM credentials, and cloud API tokens, job scheduling for recurring tasks, and a comprehensive audit trail of every playbook execution. For a bank operating across hybrid cloud (AWS EC2, on-premise VMware, Azure VMs), AAP serves as the single automation control plane. Execution environments — containerized Ansible runtimes with pre-installed collections and dependencies — ensure consistent behavior regardless of where the playbook runs.

Windows and Linux provisioning differ at the connection layer but share the same automation framework. Linux servers use SSH with key-based authentication. Windows servers use WinRM (Windows Remote Management) with certificate-based or Kerberos authentication. AAP handles this transparently through separate credential types and execution environments. A Windows execution environment includes the ansible.windows and community.windows collections, while a Linux execution environment includes ansible.posix and community.general. The key insight is that the playbook structure is identical — the difference is in the modules used (win_feature vs yum, win_service vs systemd, win_copy vs copy).

A standardized provisioning workflow in AAP uses job templates with surveys. The survey presents a form to the requester: server name, operating system (RHEL 9, Windows Server 2022, Ubuntu 22.04), environment (dev, staging, production), team, application role, and cloud target (AWS, VMware, Azure). The job template triggers a workflow that provisions the VM (calling Terraform or the cloud API), waits for the server to become reachable, then runs the appropriate provisioning playbook based on the OS. The Linux playbook applies CIS benchmark hardening (firewall rules, password policies, auditd configuration, SSH hardening), installs monitoring agents (Datadog, Splunk forwarder), configures time synchronization, joins the server to FreeIPA for centralized authentication, and registers the server in the CMDB. The Windows playbook applies STIG hardening, installs monitoring agents, joins the Active Directory domain, configures Windows Firewall rules, enables PowerShell logging for audit compliance, and registers in SCCM.

In production at a bank, AAP integrates with the ITSM workflow. A developer requests a server through ServiceNow, which triggers the AAP API via webhook. AAP provisions and configures the server, then updates the ServiceNow ticket with the server details, IP address, and compliance scan results. The entire process — from ticket creation to a fully hardened, monitored, domain-joined server — takes 15-20 minutes instead of the 2-3 days it took with manual processes. For compliance, every AAP job execution is logged with the user who triggered it, the playbook that ran, the inventory targets, and the full output. These logs feed into Splunk for SIEM integration and are available for SOX and PCI-DSS audit reviews.

The biggest gotcha is credential management across hybrid cloud. AAP's credential types handle this — you create separate credentials for AWS (access key + secret key or IAM role assumption), VMware (vCenter username/password), Azure (service principal), Linux SSH (private key), and Windows WinRM (domain admin or local admin). These credentials are encrypted at rest in AAP's database and injected into playbook executions as environment variables — operators can use credentials without seeing them. Another gotcha is execution environment bloat — do not create one massive execution environment with every collection installed. Build purpose-specific environments (windows-provisioning, linux-hardening, cloud-networking) that are small, fast to pull, and easy to maintain. Finally, test playbooks with Molecule against both fresh and partially-configured servers to ensure idempotency — a playbook should produce the same result whether it runs on a brand-new server or one that was partially provisioned before a failure.

Strategy 1: Per-Environment Workspaces (terraform workspace) Use terraform workspace new dev/staging/prod with environment-specific .tfvars files. Same code, same state backend, different workspace-isolated state files. - Pros: DRY code, easy to promote changes through environments - Cons: All environments share the same backend permissions, accidental terraform destroy in wrong workspace is catastrophic, can't have different module versions per environment - Anti-pattern: Using terraform.workspace in conditionals creates implicit differences between environments that are hard to track

Strategy 2: Per-Component Workspaces Each infrastructure component (networking, compute, data) gets its own workspace. Promotes independent lifecycle management. - Pros: Small blast radius, fast plan/apply, team ownership boundaries - Cons: Cross-workspace dependencies require data sources or remote state, ordering of applies matters

Strategy 3: Directory-Based Separation Separate directories: environments/dev/, environments/staging/, environments/prod/, each with their own backend config and state. - Pros: Maximum isolation, different module versions per environment, independent credentials - Cons: Code duplication, drift between environments, harder to promote changes

Think of Terraform's state file like a librarian's card catalog. The catalog says exactly which books are on which shelf. If someone walks in and rearranges a shelf without telling the librarian, the catalog is now wrong — not the shelf. The librarian's job on the next inventory check is to notice the mismatch and either update the catalog or put the books back where the catalog says they belong.

Terraform was designed around the idea that the .tf files are the single source of truth, and the state file is a cached snapshot of what Terraform believes exists in the real world. A manual console change never updates that cached snapshot, so the two diverge. This is intentional: Terraform does not constantly poll AWS in real time, because that would be slow and expensive at scale, so it only reconciles differences when you explicitly run a plan or apply.

Internally, terraform plan first performs a refresh: it calls the AWS API for every resource tracked in state (in this case, describe-security-groups) and compares the live attributes against what's stored in state. When it finds the ingress or egress rule block doesn't match, it marks that resource as having a diff. Because security group rules are updated in place via the AWS API (authorize/revoke operations), Terraform generates an update-in-place plan, not a destroy-and-recreate plan — so running apply is safe and reversible, it will not tear down the security group or the resources attached to it.

At production scale, this scenario is exactly why teams enable drift detection dashboards (terraform plan run on a schedule, or tools like driftctl) and lock down console write access via IAM policies or SCPs, so that engineers cannot fat-finger a manual change during an incident and forget to backport it into code. Some teams pipe scheduled drift-detection plan output into Slack so nobody discovers a silent revert three weeks later during an unrelated deploy.

The non-obvious gotcha: if the manual change was actually the correct fix (say, the developer opened port 443 during a live incident to unblock a partner integration), running terraform apply without updating the .tf file first will silently revert that fix and could reopen the incident. Always diff the plan output carefully before applying — a clean 'this brings your infrastructure in line with configuration' plan can mean 'this undoes an emergency fix' just as easily as 'this reverts an accidental change.'

Local state doesn't work for teams — two applies race. A remote backend stores state centrally and, with a lock (DynamoDB for S3, or native locking in Terraform Cloud/GCS), serializes operations so only one apply mutates state at a time. It also enables encryption and versioning.

A module bundles related resources with input variables and outputs, so you define (say) a VPC once and instantiate it per environment. Modules encapsulate complexity, enforce standards, and enable DRY infrastructure. Pin module versions from a registry for stability.

Import only updates state — it does not generate config, so you must write the resource block (or use Terraform 1.5+ import blocks / -generate-config-out) to match reality, then run plan until it shows no changes. This is how you adopt manually-created infrastructure into IaC without downtime.

Remote backends (S3+DynamoDB, Terraform Cloud, GCS) lock state to prevent concurrent applies corrupting it. The error prints the lock ID, who holds it, and when. First confirm no real apply is running (CI job, teammate). Only then terraform force-unlock <lock-id>. Prevent it by serializing applies in CI and never Ctrl-C-ing an apply mid-run.

Backend errors: check the bucket/container exists, the credentials and IAM permissions are correct, and network access to the backend — then terraform init. Module errors: verify the module source (local path, Git URL, or registry address) and the pinned version, ensure you can reach the registry/Git, and run terraform init to download it. Both are resolved at init time, not plan/apply.