Design a safe rollout strategy for a new model version serving live traffic, including rollback.
Quick Answer
Roll out in stages — shadow traffic first, then a small canary slice compared against the current model on both system and business metrics, then progressive traffic ramp-up with automated rollback triggers — and always keep the previous model version hot and instantly routable.
Detailed Answer
Start with shadow deployment: the new model receives a copy of live traffic and produces predictions that are logged but never returned to users, so you can compare its outputs against the current model with zero user-facing risk. If shadow metrics look healthy (latency, error rate, prediction distribution sanity), move to a canary: route a small percentage (e.g., 1-5%) of real traffic to the new model and compare both system metrics and downstream business metrics (conversion, false-positive rate, whatever the model drives) against the control group using a proper statistical comparison, not just eyeballing.
Ramp traffic progressively (5% → 25% → 50% → 100%) with automated guardrails that halt or roll back the rollout if key metrics regress beyond a defined threshold — this should be automatic, not dependent on someone watching a dashboard. Keep the previous model version deployed and warm (not just an artifact in the registry) so rollback is a routing change, not a redeploy-and-wait. Log which model version served every prediction so any incident can be traced back precisely, and treat the rollout itself as an experiment with a clear success/failure decision criterion defined before it starts.
Code Example
# Progressive rollout with automatic rollback guard (pseudocode)
stages = [0.01, 0.05, 0.25, 0.50, 1.0]
for pct in stages:
route_traffic(model="v2", percent=pct)
wait(minutes=30)
metrics = compare_metrics(control="v1", treatment="v2")
if metrics.error_rate_delta > THRESHOLD or metrics.business_metric_regressed:
route_traffic(model="v1", percent=1.0) # instant rollback
alert("rollout halted: regression detected at %d%% traffic" % (pct * 100))
breakInterview Tip
Emphasize that rollback needs to be a routing change with the old model already warm — "redeploy the old version" is too slow to count as a real rollback plan.