How do you configure self-hosted agents on Linux/Windows, and how do agent pools and capabilities work?

Think of self-hosted agents like a fleet of specialized delivery trucks in a logistics company. Microsoft-hosted agents are like rental trucks — convenient and maintenance-free but generic and you pay per trip. Self-hosted agents are trucks you own — they require maintenance (OS patching, tool updates) but can be customized with special equipment (GPUs, proprietary SDKs, network access to internal systems) and are always available without rental queues. Agent pools are the dispatch centers that route delivery jobs to the right truck based on what equipment the job needs.

Self-hosted agents serve three primary use cases: accessing resources behind private networks that Microsoft-hosted agents cannot reach (internal databases, private registries, on-premises servers), running builds that require specialized hardware or software not available on hosted images (GPUs for ML training, licensed tools like proprietary compilers), and reducing costs for high-volume pipelines where the per-minute hosted agent cost exceeds the cost of maintaining dedicated machines. The agent software is cross-platform, supporting Linux, Windows, and macOS.

Installation on Linux involves downloading the agent tarball, extracting it, running the configuration script with your Azure DevOps organization URL and a Personal Access Token (or service principal credentials), and installing it as a systemd service for automatic startup. The configuration script prompts for the agent pool to join, the agent name, and optionally a work directory path. For production deployments, you script this configuration non-interactively using environment variables or the --unattended flag, enabling automated provisioning through tools like Ansible or Terraform. The agent runs as a systemd service that polls Azure DevOps for queued jobs.

Capabilities are key-value pairs that describe what an agent can do. System capabilities are automatically detected: installed software, environment variables, OS version, and hardware specs. User capabilities are custom key-value pairs you add manually to describe additional tooling or attributes like team ownership, network zone, or license availability. Pipelines specify demands that match against capabilities: if a pipeline demands 'docker' and 'java17', only agents with both capabilities in their registered set will pick up the job. This prevents a Python-only agent from receiving a Java build job and failing.

Agent pools organize agents into logical groups. The default pool contains all agents, but best practice is creating separate pools for different workload types: a build-agents pool with high-CPU machines for compilation, a deploy-agents pool with network access to production, and a gpu-agents pool for ML workloads. Pool-level permissions control which pipelines can use which pools, and pool maintenance windows allow scheduling agent updates without impacting pipeline execution. Scale set agents (VMSS-backed) auto-scale the pool size based on queue depth, combining self-hosted customization with elastic capacity.

The production gotcha is agent drift and maintenance. Unlike Microsoft-hosted agents that start fresh from a golden image every run, self-hosted agents accumulate state: leftover Docker images fill disks, npm caches grow unbounded, previous build artifacts pollute the workspace, and OS packages drift from the expected versions. Teams must implement cleanup scripts (run as pipeline steps or scheduled tasks), monitor disk space and agent health, and regularly rebuild agents from automation rather than manually patching them. Another issue is agent authentication token expiration: PATs expire and must be renewed, or the agent silently stops picking up jobs. Service principal authentication with automatic credential rotation solves this.