What is continuous training (CT), and how does an ML pipeline decide when a model should retrain and when a retrained model may ship?
Quick Answer
CT extends CI/CD with automated retraining triggered by schedule, data volume, or drift signals — but the retrain trigger and the promotion decision are separate gates. Retraining is cheap and can be liberal; promotion must pass validation (data quality checks, metrics vs current champion including slices, schema/latency budgets) before any traffic shifts.
Detailed Answer
The pipeline shape: trigger (cron, N new labeled samples, drift detector firing, or upstream data-version change) -> data validation (schema, distributions, freshness, leakage checks — bad data must fail here, not train) -> training with full lineage logging (code SHA, data version, params) -> candidate evaluation against the golden set AND the live champion on overall plus slice metrics -> automated promotion gates -> staged rollout (shadow or canary) with online metric confirmation -> alias/pointer switch. The separation matters because retrain-and-auto-ship couples two decisions with different risk profiles: a scheduled retrain on quietly-corrupted upstream data will happily produce a 'successful' model that scores garbage; only the validation gates stand between that and production. Interview-worthy nuances: label lag (fraud/churn labels arrive weeks late, so 'latest data' isn't fully labeled), retrain-on-drift can amplify feedback loops (the model influences the data it will train on), and every automated gate needs a manual-override path with audit for incident response.
Code Example
# pipeline gates (pseudo-config)
trigger: {drift_psi: '>0.2', or_schedule: 'weekly'}
validate_data: [schema_match, null_rate<0.02, psi_vs_train<0.3]
promote_if:
golden_auc: '>= champion - 0.002'
slice_min_auc: '>= 0.80 # every segment'
latency_p99_ms: '< 40'
rollout: {shadow: 48h, then_canary: 10%, auto_rollback_on: online_delta<0}Interview Tip
Say 'retraining is cheap, promotion is the risk' — separating the two triggers is the design insight interviewers listen for, plus label lag as the practical complication.