Design a safe production rollout in Harness: canary deployment with Continuous Verification and automatic rollback — what are the moving parts?
Quick Answer
A canary stage ships a small instance percentage, then a CV step compares the canary's metrics/logs (from your APM — Prometheus, Datadog, etc.) against the stable baseline using ML-scored deviation; verification failure triggers the stage's rollback strategy automatically, restoring the previous verified release without a human in the loop.
Detailed Answer
Moving parts: (1) the canary deployment strategy in a CD stage — e.g., 10% -> verify -> 50% -> verify -> full, each increment a step group; (2) a health source per verification step — connectors to Prometheus/Datadog/CloudWatch/logs mapping which metrics and log queries define 'healthy' for this service (error rate, latency percentiles, exception patterns); (3) the CV analysis itself: Harness compares canary pods' signals against the current-version baseline over the analysis window (canary-vs-baseline for metrics, plus log clustering that flags novel error patterns the old version never emitted); sensitivity and duration are tunable per risk tier; (4) failure strategy on the stage: on verification failure, execute rollback — for K8s, revert to the previous release's manifests/artifact; for more complex topologies, a rollback step group you define; (5) guardrails around the automation: approval steps before prod for high-tier services, deployment freeze windows, and notification rules so rollbacks page the owning team with the CV report attached. Design points interviewers probe: baseline choice (previous version's live pods, not historical averages — traffic mix changes), analysis duration vs deploy velocity tradeoff, and ensuring the rollback path itself is exercised regularly (a never-tested rollback is a hope, not a strategy).
Code Example
execution:
steps:
- stepGroup: canary-10pct
steps:
- K8sCanaryDeploy: {instanceSelection: {count: 10, type: Percentage}}
- Verify:
type: Canary
spec: {sensitivity: HIGH, duration: 15m,
healthSources: [prometheus: error_rate, p99_latency]}
failureStrategies:
- onFailure: {errors: [Verification], action: {type: StageRollback}}Interview Tip
Name the baseline decision (canary vs current-version live pods, not historical data) and the untested-rollback trap — those two details show deployment-safety thinking beyond the product's marketing page.