Harness

Think of a Harness Delegate like a hotel concierge who works inside the building instead of a remote call-center agent trying to unlock your hotel room door from outside. The concierge (Delegate) sits inside your network, walks over to reception (your Kubernetes cluster, cloud account, Vault instance) using the keys you already gave it, and only calls the corporate office (Harness Manager, the SaaS control plane) to ask for its next instruction. Nobody outside the building ever gets a key to your rooms — the concierge always initiates the call outward, never the other way around.

Harness's SaaS control plane orchestrates pipelines (definitions of build and deploy stages) but deliberately never holds direct network access to a customer's infrastructure. This was designed for a real problem: most enterprises will not open inbound ports from a public SaaS vendor into a production VPC or on-prem data center. So Harness inverted the connection model — the Delegate, a process running as a container, pod, or VM inside the customer's network, polls the Manager over outbound HTTPS, picks up queued tasks, executes them locally with locally-scoped credentials such as a kubeconfig, cloud IAM role, or Vault token, and reports results back over that same outbound channel.

When a pipeline stage runs, the Manager doesn't reach into your cluster — it publishes a task, such as "apply this Helm chart to namespace payments-api," onto a queue tied to a Delegate selector (tags like env:prod or region:us-east-1). Any Delegate matching that selector picks up the task on its next poll cycle, executes it using the local kubeconfig or cloud SDK, streams logs back to the Manager in near real time, and acknowledges completion. If multiple Delegates share the same tags, Harness distributes work across whichever is idle, giving basic load distribution without any inbound connection ever being required.

At scale, teams typically run three or more Delegate replicas per cluster for high availability, since a single Delegate is a single point of failure for anything targeting that cluster. What to monitor: Delegate heartbeat and connectivity status in the Harness UI, Delegate pod CPU and memory (large Helm charts or big kubectl diffs can spike memory), and certificate or token expiry for the Delegate's connection to the Manager. A common failure mode is the Delegate running out of disk on its ephemeral volume from accumulated task logs, causing new tasks to fail even though the Delegate still shows "Connected."

The non-obvious gotcha is that a "Connected" Delegate in the UI only proves it can reach the Manager — it says nothing about whether it can reach the deployment target. Engineers get paged for a stuck deployment, check the Delegate page, see green, and waste twenty minutes before realizing the Delegate's outbound route to a newly peered VPC was never added to a security group. Always verify target-side connectivity, such as running kubectl get ns from inside the Delegate pod, separately from Delegate-to-Manager health.

This is like a courier who delivers a package, but the delivery confirmation text message gets lost because their phone died right after drop-off — the customer, in this case the Harness Manager, has no idea the package, your Kubernetes rollout, actually arrived, so the tracking page just spins forever showing "out for delivery."

Harness pipelines model each stage as a set of tasks it queues for a Delegate, then waits for a completion callback. This asynchronous design lets Harness support long-running deployments, such as Helm installs that take minutes, without holding an open connection the whole time, but it introduces a specific failure class: the Delegate executes the task and the underlying action genuinely finishes, yet if the Delegate process dies, gets OOMKilled, or loses network to the Manager before it reports back, Harness has no way to know the task's final outcome and leaves the stage spinning until an internal task timeout, commonly several hours by default, eventually fails it.

Diagnosing this starts by pulling the specific task ID for the stuck step from the pipeline execution details, then checking whether the Delegate that picked up that task is even still running — a Delegate restart from pod eviction, node drain, or an out-of-memory kill mid-task orphans it. Next, check kubectl rollout status or helm list directly against the target cluster to confirm the actual deployment state independent of what Harness believes. If the cluster shows the rollout complete but Harness shows running, the callback path broke; if the cluster also shows an incomplete or failed rollout, the Delegate is genuinely still working, or truly stuck on something like an image pull.

In production this pattern is the number one cause of "pipeline stuck" tickets in Harness-heavy platforms. Teams should monitor Delegate restart counts, Delegate CPU and memory utilization since undersized Delegates get OOMKilled during large Helm template renders, and set a reasonable step timeout in the pipeline YAML rather than trusting the default multi-hour timeout, so stuck stages fail fast and page someone instead of silently blocking a release train for hours.

The subtle gotcha is that manually aborting a stuck pipeline in the UI does not roll back or undo whatever the Delegate already applied to the cluster — Harness only forgets about the task, it doesn't reach back into Kubernetes to reverse the Helm release. Engineers who abort and re-run without checking cluster state end up applying the same release twice or fighting a partially-applied rollout, so always reconcile actual cluster state before retrying.

Imagine an assembly line where the only alarm is a bell that rings when a machine completely breaks down. That misses the much more common problem: a machine that's still running but producing defective parts, or one that's jammed and silently not moving anything down the line at all. A good factory floor has gauges for speed and defect rate, not just a single "broken" light.

Harness pipelines fail in three very different ways that a single "pipeline failed" notification collapses into one: a hard failure, such as a bad manifest or a failed test gate, that Harness correctly reports; a silent stall, such as a Delegate that lost connectivity or a manual approval gate nobody remembered to click, that never triggers "failed" because the pipeline is technically still "running"; and a canary rollout that Continuous Verification, or CV, Harness's automated analysis comparing canary metrics against a baseline, is quietly flagging as risky but that a human hasn't looked at yet. Each of these needs a distinct signal because they have different root causes and different urgency.

Delegate heartbeat is reported to the Manager on every poll cycle; missing several consecutive heartbeats is a leading indicator of an upcoming stall long before any pipeline actually times out. Step duration should be compared against a rolling historical baseline per step, since a deploy step for checkout-worker that usually takes ninety seconds but has been running ten minutes is a stall even though Harness hasn't marked it failed. CV risk score is calculated per verification window during a canary analysis by comparing metrics like error rate and latency between the canary and baseline pods; a rising risk score is the earliest signal of a bad deploy, well before error budgets or user-facing alerts would fire.

In practice, mature Harness setups wire these into three separate alert rules: a Delegate-down alert paging within two to three minutes of a lost heartbeat, a step-duration-anomaly alert comparing against a seven-day rolling baseline per step rather than a fixed timeout since deploy times vary by chart size and image size, and a CV-risk-score alert that pages when the automated rollback threshold is close to being crossed, so a human can intervene before the fully automatic rollback kicks in and potentially masks a real problem worth investigating.

The non-obvious gotcha is that manual approval gates look identical to a stalled Delegate from a "time since last progress" perspective — both show a pipeline sitting idle. Teams that only alert on elapsed time without distinguishing "waiting on Delegate" from "waiting on human approval" end up paging on-call engineers for approvals that were simply waiting for business hours, training people to ignore the alert entirely, which is exactly when a genuine stuck-Delegate incident gets missed.

Think of a canary deployment like a food safety inspector tasting a new recipe, the canary, against the current one, the baseline, side by side, sip by sip, over several courses — not just one bite. If the new recipe tastes slightly off on a single spoonful, a good inspector doesn't send the whole batch back; they keep tasting a few more spoonfuls to see if it's a consistent problem or just one under-salted bite.

Harness CV was designed because naive canary analysis, comparing canary error rate to baseline error rate once at one point in time, produces too many false positives — a canary pod that happens to get one slow database connection, or restarts once during rollout, looks identical to a genuinely broken release if you only look at a single snapshot. So CV instead pulls a metric time series for both canary and baseline over a rolling verification window, commonly five to ten minutes sampled every minute, and scores similarity using statistical comparison rather than a single threshold check.

In practice you configure a Verify step referencing a health source such as Prometheus or Datadog, specify the exact metric query parameterized by pod group label so it can pull canary and baseline series separately, and set the analysis to run continuously during the canary bake time. Each analysis window produces a risk score between zero and one based on how much the canary's metrics deviate from baseline; Harness accumulates these scores across consecutive windows, and only when a rolling number of windows, say three out of five, exceed the configured risk threshold does it trigger the configured on-failure action, typically an automatic rollback via a stage-rollback step.

At scale, the metric query design matters more than the threshold number. Teams commonly get burned by comparing raw counts instead of rates, since a canary with five percent of traffic will always show lower absolute error counts than baseline, which looks better and hides real problems, or by picking a verification window shorter than their p99 latency's natural noise cycle, causing constant false-positive rollbacks. What to monitor: the CV risk score trend over time per service, since a service that's frequently near-threshold even on healthy deploys means the threshold is miscalibrated for that service's normal metric variance, and rollback frequency, since an unusually high automatic-rollback rate is itself an incident worth investigating, often pointing to a monitoring or traffic-split bug rather than actual bad releases.

The non-obvious gotcha is traffic skew during a canary: if your load balancer doesn't evenly distribute traffic by request pattern, for example if all requests from one heavy customer land on the canary group by sticky session, the canary's metrics will look anomalous purely due to traffic composition, not code quality, and CV will roll back a perfectly good release. Senior engineers explicitly verify that the traffic-splitting mechanism randomizes fairly across request types before trusting CV's statistical comparison, and often exclude known-heavy or known-different traffic segments from the canary pool entirely.

The sprawl problem Harness templates solve is the same one reusable workflows solve in GitHub Actions: without them, every service clones a pipeline and they drift apart — one gets the new verification step, others don't; an incident fix lands in 12 of 50 copies. Structure: org/project-level pipeline and stage templates with version labels (v1, v2...), where services attach a thin pipeline referencing the template plus their variables; template changes are versioned so consumers upgrade deliberately (pin stable, canary the new version on low-risk services first). Input sets bundle runtime inputs (image tag, environment, feature flags) — CI triggers reference an input set rather than passing loose parameters, and overlays compose them (base + env-specific). Git Experience keeps pipeline/template/input-set YAML in Git with branch/PR semantics, giving CI-grade review and audit for delivery changes — plus environment promotion via the same PR flow as code. The governance layer on top: OPA policies (Policy as Code) can require, e.g., that prod deployments use an approved template version and include an approval stage — turning the paved road from convention into constraint.

Moving parts: (1) the canary deployment strategy in a CD stage — e.g., 10% -> verify -> 50% -> verify -> full, each increment a step group; (2) a health source per verification step — connectors to Prometheus/Datadog/CloudWatch/logs mapping which metrics and log queries define 'healthy' for this service (error rate, latency percentiles, exception patterns); (3) the CV analysis itself: Harness compares canary pods' signals against the current-version baseline over the analysis window (canary-vs-baseline for metrics, plus log clustering that flags novel error patterns the old version never emitted); sensitivity and duration are tunable per risk tier; (4) failure strategy on the stage: on verification failure, execute rollback — for K8s, revert to the previous release's manifests/artifact; for more complex topologies, a rollback step group you define; (5) guardrails around the automation: approval steps before prod for high-tier services, deployment freeze windows, and notification rules so rollbacks page the owning team with the CV report attached. Design points interviewers probe: baseline choice (previous version's live pods, not historical averages — traffic mix changes), analysis duration vs deploy velocity tradeoff, and ensuring the rollback path itself is exercised regularly (a never-tested rollback is a hope, not a strategy).