What specific Harness signals should you alert on to catch a stuck delivery pipeline before it becomes a customer-facing incident, and why is a blanket "pipeline failed" alert insufficient?

Imagine an assembly line where the only alarm is a bell that rings when a machine completely breaks down. That misses the much more common problem: a machine that's still running but producing defective parts, or one that's jammed and silently not moving anything down the line at all. A good factory floor has gauges for speed and defect rate, not just a single "broken" light.

Harness pipelines fail in three very different ways that a single "pipeline failed" notification collapses into one: a hard failure, such as a bad manifest or a failed test gate, that Harness correctly reports; a silent stall, such as a Delegate that lost connectivity or a manual approval gate nobody remembered to click, that never triggers "failed" because the pipeline is technically still "running"; and a canary rollout that Continuous Verification, or CV, Harness's automated analysis comparing canary metrics against a baseline, is quietly flagging as risky but that a human hasn't looked at yet. Each of these needs a distinct signal because they have different root causes and different urgency.

Delegate heartbeat is reported to the Manager on every poll cycle; missing several consecutive heartbeats is a leading indicator of an upcoming stall long before any pipeline actually times out. Step duration should be compared against a rolling historical baseline per step, since a deploy step for checkout-worker that usually takes ninety seconds but has been running ten minutes is a stall even though Harness hasn't marked it failed. CV risk score is calculated per verification window during a canary analysis by comparing metrics like error rate and latency between the canary and baseline pods; a rising risk score is the earliest signal of a bad deploy, well before error budgets or user-facing alerts would fire.

In practice, mature Harness setups wire these into three separate alert rules: a Delegate-down alert paging within two to three minutes of a lost heartbeat, a step-duration-anomaly alert comparing against a seven-day rolling baseline per step rather than a fixed timeout since deploy times vary by chart size and image size, and a CV-risk-score alert that pages when the automated rollback threshold is close to being crossed, so a human can intervene before the fully automatic rollback kicks in and potentially masks a real problem worth investigating.

The non-obvious gotcha is that manual approval gates look identical to a stalled Delegate from a "time since last progress" perspective — both show a pipeline sitting idle. Teams that only alert on elapsed time without distinguishing "waiting on Delegate" from "waiting on human approval" end up paging on-call engineers for approvals that were simply waiting for business hours, training people to ignore the alert entirely, which is exactly when a genuine stuck-Delegate incident gets missed.