How do you implement a Terraform CI/CD pipeline with plan approval workflow?
Quick Answer
Implement a CI/CD pipeline that runs terraform plan on pull requests, posts the plan output as a PR comment for review, requires manual approval before apply, and uses remote state locking to prevent concurrent operations. Use OIDC authentication, separate plan and apply stages, and implement policy-as-code gates with Sentinel or OPA.
Detailed Answer
A production-grade Terraform CI/CD pipeline must solve five problems: authentication without long-lived credentials, plan visibility for reviewers, approval gates before destructive changes, concurrency control to prevent conflicting applies, and policy enforcement to catch compliance violations before they reach infrastructure. Think of it like a surgical operation workflow: the surgeon (engineer) proposes an operation (plan), it gets reviewed by a board (PR review), approved by an authority (manual approval gate), and only then executed in a controlled environment (apply) with safeguards (state locking).
Authentication should use OIDC federation. GitHub Actions, GitLab CI, and CircleCI all support OIDC tokens that can be exchanged for short-lived AWS credentials via STS AssumeRoleWithWebIdentity. This eliminates the need to store AWS access keys as CI secrets, which is a common audit finding. The OIDC trust policy should be scoped to specific repositories and branches to prevent unauthorized access.
The pipeline structure typically has three stages. The first stage runs on every pull request: terraform init, terraform validate, terraform fmt -check, and terraform plan -out=plan.tfplan. The plan output is captured and posted as a PR comment using a tool like tfcmt or a custom script that parses the plan JSON output. This gives reviewers visibility into exactly what will change.
The second stage is the approval gate. For non-production environments, this might be automatic after PR merge. For production, it requires explicit manual approval. In GitHub Actions, this is implemented using environments with required reviewers. In GitLab, it is a manual job gate. The approval should be from someone other than the PR author (four-eyes principle) and ideally from a platform engineering team member who understands the blast radius.
The third stage runs terraform apply using the saved plan file. This is critical: never re-run plan during apply, because infrastructure may have changed between the plan and apply stages. The saved plan file ensures exactly what was reviewed gets applied. After apply, the pipeline should post the apply output back to the PR or a Slack channel for visibility.
Policy-as-code adds guardrails. HashiCorp Sentinel (Terraform Cloud/Enterprise) or Open Policy Agent (open source) evaluate the plan against organizational policies: no public S3 buckets, all RDS instances must have encryption, all security groups must have descriptions. These checks run after plan but before approval, catching violations early.
Production gotchas include handling plan file expiration (plan files reference specific provider plugin versions and state serial numbers, so they expire when state changes), managing workspace-level parallelism (only one pipeline should operate on a workspace at a time), and dealing with long-running applies that exceed CI timeout limits. Some teams implement a Terraform-specific lock in Redis or DynamoDB beyond the state lock, to queue pipeline runs at the workspace level.