How do you manage secrets in Flux using Mozilla SOPS encryption?

Think of SOPS encryption with Flux like a diplomatic pouch system. Diplomats (developers) write sensitive messages (Kubernetes Secrets) and seal them in tamper-proof diplomatic pouches (SOPS-encrypted files) that can safely travel through public channels (Git repositories). Only the embassy at the destination (the Flux kustomize-controller with the decryption key) can open the pouch and read the contents. Even if someone intercepts the pouch during transit (gains read access to the Git repository), they cannot read the message because it is encrypted. This allows teams to follow GitOps principles — storing everything in Git as the single source of truth — without exposing sensitive data like database passwords, API keys, and TLS certificates.

Mozilla SOPS (Secrets OPerationS) is an encryption tool that encrypts only the values in structured data files like YAML and JSON, leaving the keys and structure visible. This is a crucial distinction from tools that encrypt the entire file, because it means you can still see what a Secret contains (which keys exist, what the resource structure looks like) without being able to read the actual sensitive values. SOPS supports multiple encryption backends: AGE (a modern, simple encryption tool), PGP/GPG (the traditional approach), AWS KMS, GCP KMS, Azure Key Vault, and HashiCorp Vault Transit. In Kubernetes environments, AGE is increasingly preferred for its simplicity — a single key pair without the complexity of the PGP web of trust — while cloud KMS options are favored in managed cloud environments for centralized key management and audit logging.

To use SOPS with Flux, you first encrypt your Kubernetes Secret manifests using the sops CLI tool. You create a .sops.yaml configuration file in your repository that defines which files to encrypt and which encryption keys to use, allowing different encryption rules for different paths or file patterns. When you run sops --encrypt on a Secret YAML file, SOPS encrypts the values in the data and stringData fields while preserving the YAML structure, and adds a sops metadata block to the file containing the encrypted data key and information about which master keys can decrypt it. The encrypted file is then committed to Git. On the cluster side, you configure the Flux Kustomization with spec.decryption.provider set to sops and spec.decryption.secretRef pointing to a Kubernetes Secret containing the decryption key (for AGE, this is the private key; for cloud KMS, this is the service account credentials).

In production environments managing services like payments-api and user-auth-service, the SOPS workflow integrates into the development pipeline at several touchpoints. Developers encrypt secrets locally before committing, CI pipelines validate that no unencrypted secrets are committed by checking for the sops metadata block, and Flux decrypts them during reconciliation. A common production pattern uses cloud KMS for master key management — for example, AWS KMS with a dedicated key for each environment (development, staging, production) — and configures the .sops.yaml file to automatically select the correct KMS key based on the file path. This means secrets in the deploy/production directory are encrypted with the production KMS key, and only the production cluster's Flux instance (which has IAM credentials for that KMS key) can decrypt them. This provides cryptographic separation between environments even when using a single Git repository.

A critical production gotcha is key rotation. When you rotate the SOPS master key (whether AGE, PGP, or KMS), you must re-encrypt all existing secrets with the new key and update the decryption key secret on the cluster. SOPS supports multiple master keys simultaneously, which enables a graceful rotation process: add the new key to the .sops.yaml configuration, run sops updatekeys on all encrypted files to add the new key while keeping the old one, update the cluster decryption secret with the new key, and finally remove the old key from .sops.yaml and run updatekeys again. Another common mistake is accidentally committing unencrypted secrets — teams should implement Git pre-commit hooks that check for unencrypted Secret resources and block the commit. Additionally, be aware that SOPS encrypts values but not keys, so sensitive information should never appear in the key names of a Secret's data field. Finally, the decryption key secret itself must be bootstrapped onto the cluster outside of the GitOps workflow, typically using a sealed initial setup script or a cloud-native secret manager injection, since Flux cannot decrypt the key needed to decrypt itself.