A Harness CD pipeline shows the deployment stage stuck in "Running" for over an hour with no new log lines, while the target Kubernetes cluster shows the rollout actually completed successfully — what are the possible causes and how do you isolate them?
Quick Answer
This mismatch usually means the Delegate lost its connection back to the Harness Manager after starting the task, so the cluster-side action finished but Harness never received the completion signal. Isolate it by checking Delegate connectivity and logs first to rule in or out a lost callback, then verify actual cluster state directly with kubectl or helm rather than trusting what the pipeline UI reports.
Detailed Answer
This is like a courier who delivers a package, but the delivery confirmation text message gets lost because their phone died right after drop-off — the customer, in this case the Harness Manager, has no idea the package, your Kubernetes rollout, actually arrived, so the tracking page just spins forever showing "out for delivery."
Harness pipelines model each stage as a set of tasks it queues for a Delegate, then waits for a completion callback. This asynchronous design lets Harness support long-running deployments, such as Helm installs that take minutes, without holding an open connection the whole time, but it introduces a specific failure class: the Delegate executes the task and the underlying action genuinely finishes, yet if the Delegate process dies, gets OOMKilled, or loses network to the Manager before it reports back, Harness has no way to know the task's final outcome and leaves the stage spinning until an internal task timeout, commonly several hours by default, eventually fails it.
Diagnosing this starts by pulling the specific task ID for the stuck step from the pipeline execution details, then checking whether the Delegate that picked up that task is even still running — a Delegate restart from pod eviction, node drain, or an out-of-memory kill mid-task orphans it. Next, check kubectl rollout status or helm list directly against the target cluster to confirm the actual deployment state independent of what Harness believes. If the cluster shows the rollout complete but Harness shows running, the callback path broke; if the cluster also shows an incomplete or failed rollout, the Delegate is genuinely still working, or truly stuck on something like an image pull.
In production this pattern is the number one cause of "pipeline stuck" tickets in Harness-heavy platforms. Teams should monitor Delegate restart counts, Delegate CPU and memory utilization since undersized Delegates get OOMKilled during large Helm template renders, and set a reasonable step timeout in the pipeline YAML rather than trusting the default multi-hour timeout, so stuck stages fail fast and page someone instead of silently blocking a release train for hours.
The subtle gotcha is that manually aborting a stuck pipeline in the UI does not roll back or undo whatever the Delegate already applied to the cluster — Harness only forgets about the task, it doesn't reach back into Kubernetes to reverse the Helm release. Engineers who abort and re-run without checking cluster state end up applying the same release twice or fighting a partially-applied rollout, so always reconcile actual cluster state before retrying.