The Jenkins build queue is growing and jobs are waiting a long time to start even though the master itself looks healthy. How would you diagnose this?

The Build Queue view (and /computer/api/json or the Jenkins CLI) shows exactly which jobs are queued and, critically, the reason each is waiting — 'Waiting for next available executor' versus a specific message about label restrictions not being satisfiable by any currently-connected agent. This distinction matters: if it's a raw executor shortage, the fix is more agents (static pool size or dynamic agent scaling limits); if it's a label mismatch (a job pinned to agent { label 'gpu-build' } and no GPU-labeled agent is currently online), adding more generic agents does nothing — you need agents matching that specific label.

Separately, check the Jenkins controller's own health — heavy use of Scripted Pipeline or complex Declarative pipelines with many parallel branches puts real CPU/memory load on the controller itself (Jenkins pipelines use Groovy CPS transformation and checkpoint their execution state, which is more expensive than it looks for very large or highly parallel pipelines), and a controller under memory pressure can slow down scheduling decisions for every job, not just specific ones, even while individual builds that DO get an agent run fine.

Dynamic agent provisioning issues are a third category: if using cloud/Kubernetes dynamic agents, check whether the underlying cluster/cloud API is itself rate-limited, out of capacity, or slow to schedule pods — this shows up as queued jobs waiting specifically on 'agent is being provisioned' rather than a pure executor-count problem, and the fix lives in the underlying infrastructure, not Jenkins configuration.