You support 30 customer deployments of your product, each with environment-specific configuration. How do you prevent config drift from eating the team alive?
Quick Answer
Golden base configuration in version control, per-customer overlays containing only their deltas, rendered and validated in CI — never hand-edited files in customer environments. Drift detection compares running config against the rendered source of truth, and every emergency hand-edit gets backported into the overlay the same week.
Detailed Answer
The failure mode without discipline: each deployment accumulates undocumented hand-edits (that one customer's proxy, another's cipher requirements, a hotfix flag from an incident two years ago nobody remembers), until upgrades become bespoke archaeology per customer. The structure that works: (1) a golden config — the product's supported baseline, versioned with the release; (2) per-customer overlay repos/directories containing only justified deltas, each with a comment linking why (ticket/constraint); (3) a render step (kustomize/helm values layering/jinja) producing the effective config in CI, with schema validation and policy checks (e.g., 'TLS floor never lowered'); (4) deployment only from rendered artifacts — direct edits in the environment are, at most, incident stopgaps with a mandatory backport ticket; (5) drift detection: a periodic job diffs live config against the rendered truth and alarms on unexplained deltas. Upgrades then become: bump golden version, re-render every customer, review the diff per customer in minutes instead of re-discovering their environment. The cultural half: making the backport-after-emergency loop actually close is the FDE lead's job — unbackported hotfixes are where drift is born.
Code Example
customers/
_golden/values.yaml # product baseline, versioned
acme/values.yaml # deltas only, each annotated:
# proxy: required — ACME egress via squid (TICKET-812)
# tls_min: 1.3 — customer policy (TICKET-901)
CI: render(golden + overlay) -> schema check -> policy check -> artifact
drift job: diff(live, artifact) -> alert on unexplained deltaInterview Tip
The two details that land: overlays contain deltas only (each annotated with why), and the emergency-edit-then-backport loop with enforcement — drift is a process failure before it's a tooling failure.