MLOps

Traditional DevOps assumes that the same code plus the same config produces the same behavior forever, and that testing catches most regressions before release. ML systems break that assumption in two ways: the model's behavior depends on training data that changes over time, and a model can silently degrade in production even with unchanged code, because the real-world data it sees starts to differ from what it was trained on.

MLOps extends the standard pipeline with: dataset versioning (so you can reproduce exactly what a model was trained on), experiment tracking (hyperparameters, metrics, artifacts per training run), a model registry with staged promotion (dev → staging → production), automated evaluation gates before deployment, and continuous monitoring for both system health (latency, errors) and model health (prediction drift, data drift, accuracy decay). The deployment unit is a model artifact plus its serving code, not just code.

There are two related but distinct problems: data drift (the distribution of incoming features changes) and concept drift (the relationship between features and the true label changes, even if feature distributions look stable). Data drift is detectable immediately by comparing live feature statistics against a reference window using tests like population stability index (PSI), KL divergence, or simple summary-statistic monitoring per feature. Concept drift usually requires delayed ground truth (e.g., did the fraud prediction turn out to be correct once a chargeback occurred), so you track a lagging accuracy/precision/recall metric alongside the drift signals.

When drift crosses a threshold, the response depends on cause: if a specific feature's upstream data source changed shape (schema change, new user segment, an outage that shifted traffic mix), fix the pipeline. If the world genuinely changed (seasonality, new product usage patterns), trigger a retraining job on fresher data. If confidence in the current model is low, fall back to a simpler rules-based system or a previous model version while retraining/investigation happens — treat this like an incident with a runbook, not an ad hoc fire drill.

Without a feature store, teams typically write feature logic twice: once in a batch/offline pipeline (e.g., Spark/SQL) to build training datasets, and once in an online service (e.g., application code) to compute features at request time for real-time predictions. Any divergence between those two implementations — different time windows, different null-handling, different rounding, a bug fixed in one but not the other — produces train/serve skew: the model performs well offline but degrades in production because it's effectively seeing out-of-distribution inputs.

A feature store addresses this by defining feature transformation logic once and materializing it to two stores: an offline store (historical, point-in-time correct, used for training) and an online store (low-latency key-value lookup, used for serving), both fed from the same feature definitions. Point-in-time correctness matters specifically to avoid label leakage — when building training data, you must look up feature values as they existed at the time of the historical event, not the current value. Good feature stores also version feature definitions and track lineage, so you can tell exactly which feature version a given model was trained against.

Start with shadow deployment: the new model receives a copy of live traffic and produces predictions that are logged but never returned to users, so you can compare its outputs against the current model with zero user-facing risk. If shadow metrics look healthy (latency, error rate, prediction distribution sanity), move to a canary: route a small percentage (e.g., 1-5%) of real traffic to the new model and compare both system metrics and downstream business metrics (conversion, false-positive rate, whatever the model drives) against the control group using a proper statistical comparison, not just eyeballing.

Ramp traffic progressively (5% → 25% → 50% → 100%) with automated guardrails that halt or roll back the rollout if key metrics regress beyond a defined threshold — this should be automatic, not dependent on someone watching a dashboard. Keep the previous model version deployed and warm (not just an artifact in the registry) so rollback is a routing change, not a redeploy-and-wait. Log which model version served every prediction so any incident can be traced back precisely, and treat the rollout itself as an experiment with a clear success/failure decision criterion defined before it starts.

The pipeline shape: trigger (cron, N new labeled samples, drift detector firing, or upstream data-version change) -> data validation (schema, distributions, freshness, leakage checks — bad data must fail here, not train) -> training with full lineage logging (code SHA, data version, params) -> candidate evaluation against the golden set AND the live champion on overall plus slice metrics -> automated promotion gates -> staged rollout (shadow or canary) with online metric confirmation -> alias/pointer switch. The separation matters because retrain-and-auto-ship couples two decisions with different risk profiles: a scheduled retrain on quietly-corrupted upstream data will happily produce a 'successful' model that scores garbage; only the validation gates stand between that and production. Interview-worthy nuances: label lag (fraud/churn labels arrive weeks late, so 'latest data' isn't fully labeled), retrain-on-drift can amplify feedback loops (the model influences the data it will train on), and every automated gate needs a manual-override path with audit for incident response.

The failure that motivates it: a model regresses, you retrain from 'the same' code, and get a different model — because the feature table mutated in place. Approaches: (1) content-addressed pipelines (DVC/lakeFS) — data files hashed, referenced by pointer files that live in git, so git checkout reconstructs the exact dataset; (2) lakehouse time travel (Delta/Iceberg table versions) — training reads a pinned snapshot ID, cheap because storage is shared across versions; (3) discipline-level minimum: immutable dated partitions plus a manifest (paths + checksums + row counts) logged with the run. Whichever mechanism, the contract is the same: the training run's metadata records an immutable data reference, and re-running with that reference is byte-identical. This also unlocks: auditability (regulators ask 'what data trained this?'), safe backfills (new dataset version, not in-place mutation), and debugging skew (diff data versions between good and bad models). The trap to name: versioning raw data but not the feature-engineering output — if features are computed by a mutable pipeline at train time, pin the feature table, not just the source.

What transfers directly: treat every behavior-changing artifact as versioned and gated — except now those artifacts are prompts, chat templates, retrieval parameters, tool definitions, and the provider model ID (which vendors update under you; pin dated snapshots where offered). Evaluation changes character: classical ML has ground-truth labels and crisp metrics; LLM outputs need golden sets scored by rubric-driven LLM judges calibrated against periodic human review, plus hard checks where possible (schema validation, citation presence, safety filters). Monitoring adds new axes: token spend per request/tenant (a cost regression is a production incident), time-to-first-token, refusal/fallback rates, and quality sampling on live traffic. New failure modes with no classical analog: prompt injection, context-window truncation, provider-side model updates changing behavior with zero deploys on your side, and non-determinism making 'reproduce the bug' probabilistic. What breaks: drift detection on input distributions means little when inputs are free text — you monitor output quality and task success instead. The punchline: the governance instinct is identical, the artifact list and eval machinery are new.