How do you debug failing Tekton TaskRuns and PipelineRuns?

Think of debugging Tekton pipelines like troubleshooting a car that will not start. You do not immediately tear apart the engine. Instead, you follow a systematic diagnostic checklist: check fuel, check battery, check starter motor, check spark plugs. Each check narrows down the problem. In Tekton, you follow a similar layered approach: start with the PipelineRun status, drill into the specific failed TaskRun, examine the step container logs, and inspect the underlying pod events. This top-down methodology prevents wasting time on irrelevant components.

The first diagnostic step is examining the PipelineRun or TaskRun status using kubectl describe or tkn pipelinerun describe. The status section contains a conditions array with the overall run status, a childReferences or taskRuns map showing the status of each individual Task, and timestamps for when each Task started and completed. A PipelineRun can fail for several reasons: a Task step exited with a non-zero exit code, a when expression evaluated to false causing unexpected skips, a timeout was exceeded, or a workspace binding failed. The tkn pipelinerun describe command provides a human-readable summary showing which Tasks succeeded with a checkmark, which failed with an X, which were skipped, and which timed out. For the payments-api pipeline, if the security-scan Task failed, this summary immediately tells you where to focus your investigation.

Drilling into the failed TaskRun reveals step-level details. Each Task can contain multiple steps, and Tekton tracks the exit code and termination reason for every step container. The tkn taskrun logs command streams the logs from all step containers in execution order, or you can target a specific step with the --step flag. For long-running Tasks in the order-processing-service pipeline, filtering logs to the failing step avoids scrolling through megabytes of irrelevant output. If the step container was OOMKilled, the termination reason in the TaskRun status will indicate this, pointing to a resource limits issue rather than a code bug. If the step exited with a specific error code, the container logs usually contain the stack trace or error message that explains the failure.

Pod-level issues represent a separate class of failures that manifest before any step container even starts. Common pod-level problems include ImagePullBackOff when the step container image cannot be pulled from the registry, pending pods due to insufficient cluster resources or node affinity rules that cannot be satisfied, PVC binding failures when the workspace volume cannot be provisioned, and service account permission errors when the TaskRun SA lacks RBAC permissions to create or access resources. Use kubectl describe pod on the TaskRun pod to inspect the events section, which contains detailed messages about scheduling decisions, image pulls, volume mounts, and container starts. For the user-auth-service pipeline, an ImagePullBackOff on the custom test runner image indicates either incorrect image tags, missing registry credentials, or a registry outage, and the pod events will specify which.

Advanced debugging techniques include using the Tekton Dashboard for visual pipeline execution tracing, which shows the DAG of Tasks with color-coded statuses and clickable log viewers. For intermittent failures, examine the TaskRun's retries field if retry policies are configured, and compare timestamps between the failing run and surrounding successful runs to identify environmental changes. Enable verbose logging in the Tekton controller by modifying the config-logging ConfigMap in the tekton-pipelines namespace, which provides detailed information about Pipeline validation, Task scheduling, and result propagation. For workspace-related failures, exec into a debug pod using the same PVC to inspect file permissions, disk space, and directory structure. Always check whether the failure is deterministic by re-running the PipelineRun with the same parameters, as transient failures from network timeouts, registry rate limits, or resource pressure require different remediation than code-level bugs.