How do you detect and respond to model drift in a production ML pipeline?
Quick Answer
Continuously compare live input feature distributions and prediction distributions against a reference (typically the training set), alert when they diverge beyond a threshold, and separately track ground-truth performance where labels eventually become available — then respond with retraining, feature fixes, or rollback depending on what the drift indicates.
Detailed Answer
There are two related but distinct problems: data drift (the distribution of incoming features changes) and concept drift (the relationship between features and the true label changes, even if feature distributions look stable). Data drift is detectable immediately by comparing live feature statistics against a reference window using tests like population stability index (PSI), KL divergence, or simple summary-statistic monitoring per feature. Concept drift usually requires delayed ground truth (e.g., did the fraud prediction turn out to be correct once a chargeback occurred), so you track a lagging accuracy/precision/recall metric alongside the drift signals.
When drift crosses a threshold, the response depends on cause: if a specific feature's upstream data source changed shape (schema change, new user segment, an outage that shifted traffic mix), fix the pipeline. If the world genuinely changed (seasonality, new product usage patterns), trigger a retraining job on fresher data. If confidence in the current model is low, fall back to a simpler rules-based system or a previous model version while retraining/investigation happens — treat this like an incident with a runbook, not an ad hoc fire drill.
Code Example
# Simple feature drift check (population stability index)
def psi(reference: np.ndarray, current: np.ndarray, bins: int = 10) -> float:
breakpoints = np.percentile(reference, np.linspace(0, 100, bins + 1))
ref_pct = np.histogram(reference, breakpoints)[0] / len(reference)
cur_pct = np.histogram(current, breakpoints)[0] / len(current)
ref_pct, cur_pct = np.clip(ref_pct, 1e-6, None), np.clip(cur_pct, 1e-6, None)
return float(np.sum((cur_pct - ref_pct) * np.log(cur_pct / ref_pct)))
# PSI > 0.2 on a key feature is a common trigger for investigation/retrainingInterview Tip
Distinguish data drift from concept drift explicitly — interviewers use this question to check whether you conflate the two.