How does Flux variable substitution work in Kustomizations for environment-specific configurations?

Think of Flux variable substitution like a mail merge for your Kubernetes manifests. You write a template letter (your base YAML manifests) with placeholders like Dear ${CUSTOMER_NAME}, and the merge engine (Flux postBuild substitution) fills in the actual values from a data source (a ConfigMap or inline map) before sending the letter (applying to the cluster). This allows you to maintain a single set of manifests that adapts to different environments — development, staging, production — by simply changing the variable values rather than maintaining separate copies of the same YAML files with minor differences.

Flux implements variable substitution through the spec.postBuild section of a Kustomization resource. The postBuild processing happens after Kustomize has rendered the final manifests but before they are applied to the cluster. This means the substitution works on the fully rendered output, including any overlays, patches, and transformations that Kustomize has already applied. The substitution engine scans the rendered manifests for patterns matching ${VAR_NAME} and replaces them with the corresponding values. Two sources of variables are supported: spec.postBuild.substitute provides an inline key-value map directly in the Kustomization spec, and spec.postBuild.substituteFrom references external Kubernetes ConfigMaps or Secrets that contain the variable values. When both are used, inline substitute values take precedence over substituteFrom values, allowing you to override specific variables while inheriting the rest from a shared ConfigMap.

The variable substitution syntax supports a default value fallback using the pattern ${VAR_NAME:-default_value}, which provides a value to use if the variable is not defined in any source. This is particularly useful for optional configuration that should have sensible defaults. Variables can be used anywhere in the rendered manifests — in metadata like labels and annotations, in spec fields like replica counts and resource limits, in container image tags, in environment variables, and even in multi-line string values like ConfigMap data. However, the substitution is purely string-based, so it does not validate the resulting YAML structure. If a substitution produces invalid YAML — for example, replacing a numeric field with a string — the reconciliation will fail during the apply phase with a validation error.

In production environments, teams typically use a layered substitution strategy for services like payments-api and inventory-service. A base Kustomization defines the application manifests with placeholders for environment-specific values like ${CLUSTER_NAME}, ${ENVIRONMENT}, ${REPLICAS}, ${CPU_LIMIT}, and ${LOG_LEVEL}. A ConfigMap named cluster-vars in the flux-system namespace contains cluster-wide values like the cluster name and environment label. A separate ConfigMap named payments-vars contains service-specific values like replica counts and resource limits. The Kustomization's substituteFrom field references both ConfigMaps, with the service-specific one listed last to allow it to override any conflicting keys from the cluster-wide ConfigMap. This pattern enables platform teams to manage global configuration centrally while giving application teams control over their service-specific parameters.

A critical gotcha is that variable substitution does not work with Kustomize's built-in variable replacement mechanism (vars/replacements) — these are two separate systems. Flux postBuild substitution happens after Kustomize renders, while Kustomize vars are resolved during rendering. Mixing them can lead to confusion about which system resolves which placeholders. Another common pitfall is security-related: if you use substituteFrom with a Secret, the secret values are embedded directly into the rendered manifests, which means they may appear in plain text in logs, events, or the Flux source artifact. For sensitive values like database passwords, you should use Kubernetes Secrets mounted as volumes or environment variables in your pod spec rather than substituting them into the manifest text. Teams should also be aware that undefined variables without defaults are left as-is in the rendered output — ${UNDEFINED_VAR} will literally appear as the string ${UNDEFINED_VAR} in the applied manifest, which can cause subtle deployment issues that are hard to debug without inspecting the rendered output.