How do you debug failing Azure DevOps pipelines — what logs, system variables, and diagnostic tools do you use?

Think of debugging a failing pipeline like diagnosing a car that intermittently stalls. You would not just look at the engine — you would check the dashboard gauges (system variables), review the maintenance log (pipeline logs), enable the onboard diagnostics mode (system.debug), check the computer's error codes (agent diagnostic logs), and compare recent trips where the car worked versus stalled (passing vs failing runs). Each diagnostic tool reveals a different layer of the problem, and complex failures require combining multiple sources.

The first debugging step is enabling verbose logging by setting the system.debug variable to true. You can set this at queue time when manually running the pipeline, or add it to the variables section temporarily. With debug enabled, every task outputs dramatically more information: HTTP request/response details for API calls, full command-line arguments for scripts, environment variable listings, file download progress, and internal task decision logic. This verbose output often immediately reveals the issue — a 403 authentication error buried in a generic 'task failed' message, or a file path that resolves differently than expected.

System variables provide context about the pipeline execution environment. Build.Reason tells you why the pipeline triggered (IndividualCI, PullRequest, Manual, Schedule), which matters when behavior differs between trigger types. Agent.OS and Agent.OSArchitecture reveal the execution platform. Build.SourceBranch and Build.SourceVersion identify exactly which code version is building. System.JobAttempt shows retry count for re-run scenarios. Adding a diagnostic step that prints key system variables at the start of every job helps establish the execution context when reviewing failed runs later.

Agent diagnostic logs provide low-level information unavailable in pipeline logs. On self-hosted agents, the _diag directory contains Worker and Agent log files with details about job acquisition, capability evaluation, task download, and resource constraints. These logs reveal issues like disk space exhaustion (agent cannot download tasks), network connectivity problems (agent cannot reach Azure DevOps), or capability mismatches (job demands do not match agent capabilities). For Microsoft-hosted agents, some of this information surfaces in the pipeline log's initialization steps, but agent-level logs are not directly accessible.

The Timeline API provides structured data about each step's duration, result, and error messages. Querying the timeline for a failed run (GET _apis/build/builds/{buildId}/timeline) returns JSON with each record's state, result, start/finish times, and issues array. This structured data enables automated analysis: comparing step durations between passing and failing runs to identify timing-related failures, or extracting specific error messages programmatically. For intermittent failures, scripting timeline API queries across multiple runs can reveal patterns invisible from examining individual run logs.

Custom diagnostic techniques include strategic use of logging commands (##vso[task.logissue] for warnings/errors that surface in the run summary), output variables to pass diagnostic data between steps, and conditional diagnostic steps that only run on failure (condition: failed()). For network-related issues, adding curl or wget test steps that verify connectivity to external services before the actual task runs isolates whether the failure is a network issue or a logic issue. For file-related issues, find and ls commands before the failing step reveal the actual filesystem state.

The production gotcha is debugging intermittent failures — problems that occur once every 10 runs with no obvious pattern. These are typically caused by infrastructure flakiness (hosted agent image updates, transient network issues, dependency registry rate limiting), timing races (tests depending on clock time, steps assuming previous async operations completed), or resource exhaustion (parallel jobs consuming all available memory on a shared agent). For these, enable pipeline retention to preserve logs from failed runs, implement structured logging that includes timestamps and request IDs, and add retry logic with the retryCountOnTaskFailure setting while you investigate the root cause.