How do you handle Terraform state locking and what happens during lock conflicts?

Terraform state locking is a concurrency control mechanism that prevents two or more operators from writing to the same state file simultaneously, which would cause state corruption and potentially orphaned cloud resources. Think of it like a database row-level lock: before Terraform can modify state, it must first acquire an exclusive lock, and any other process attempting to modify the same state must wait or fail.

When you run terraform apply or terraform plan (with certain backends), Terraform sends a Lock request to the backend. For S3+DynamoDB, this means writing a record to the DynamoDB table with the state file's digest as the partition key and a unique LockID. The LockID contains the operator's hostname, the Terraform operation type, the workspace name, and a timestamp. DynamoDB's conditional write ensures atomicity: if a record already exists with that key, the write fails with a ConditionalCheckFailedException, and Terraform reports that the state is locked by another process.

In production, lock conflicts arise in several scenarios. The most common is when two engineers run terraform apply on the same workspace concurrently. Terraform will display the lock holder's information including their username, operation type, and when the lock was acquired. The blocked operator must wait for the first operation to complete. Another common scenario is a CI/CD pipeline crash: if a pipeline runner dies mid-apply, the lock remains in DynamoDB as a stale lock. This requires manual intervention using terraform force-unlock with the lock ID.

Force-unlock is dangerous in production because you cannot guarantee the previous operation completed cleanly. Before force-unlocking, you should verify the state of the actual infrastructure using AWS console or CLI, check the DynamoDB table directly to confirm the lock metadata, and review CloudTrail logs for any API calls made by the crashed process. A safer pattern is to implement lock timeouts in your CI/CD pipeline: wrap terraform apply in a timeout command and have the pipeline explicitly run terraform force-unlock only after confirming no infrastructure changes are in progress.

Different backends handle locking differently. Consul uses its built-in session-based locking with TTL. Azure Blob Storage uses native blob leases. Google Cloud Storage uses object generation numbers for optimistic locking. Not all backends support locking; the local backend does via filesystem locks, but NFS-mounted local backends have notoriously unreliable locking, which is why teams migrate to remote backends. The etcd backend uses its compare-and-swap primitive. Understanding your backend's locking semantics is critical for disaster recovery planning, because a corrupted lock table can block all infrastructure changes across your organization.