intermediategeneral

How do you implement Jenkins agents/workers for distributed builds?

⚡

Quick Answer

Jenkins agents are separate machines or containers that connect to the Jenkins controller to execute build jobs, enabling distributed builds that scale horizontally by offloading work from the controller and running multiple builds in parallel across different environments.

Detailed Answer

Think of a Jenkins distributed build setup like a restaurant kitchen. The head chef (Jenkins controller) plans the menu, takes orders, and coordinates the workflow, but does not actually cook. The line cooks (agents) do the actual preparation and cooking at their individual stations. Each station might specialize in different things: one handles grilling, another handles pastry, and another handles salads. By adding more line cooks, the restaurant can handle more orders simultaneously without the head chef becoming a bottleneck. Similarly, Jenkins agents handle the actual build execution while the controller focuses on orchestration.

Jenkins agents connect to the controller using one of several communication methods. The most common approaches are SSH-based agents where the controller connects to the agent machine via SSH, JNLP agents where the agent initiates an outbound connection to the controller, and Kubernetes-based agents where pods are dynamically provisioned for each build. Each agent is configured with labels that describe its capabilities, such as the operating system, installed tools, or hardware specifications. Pipeline jobs use the agent directive to request an agent with specific labels, and the Jenkins scheduler matches jobs to available agents based on these labels and current load.

Internally, when a pipeline requires an agent, the Jenkins remoting layer establishes a bidirectional communication channel between the controller and the agent. The controller sends the build steps to the agent for execution, and the agent streams back console output, test results, and build artifacts. The workspace directory on the agent holds the checked-out source code and build outputs, and Jenkins manages workspace allocation to prevent conflicts between concurrent builds. The remoting protocol handles serialization of Java objects between the controller and agent JVMs, which is why agent and controller Jenkins versions should be kept in sync to avoid compatibility issues.

In production environments, teams typically run a pool of static agents for predictable baseline capacity and supplement them with dynamic agents that scale based on demand. Cloud plugins for AWS, Azure, GCP, and Kubernetes allow Jenkins to automatically provision agent machines when the build queue grows and terminate them when idle. Docker-based agents provide clean, reproducible build environments by starting each build in a fresh container with pre-installed tools. Labels should follow a consistent naming convention that reflects both capability and environment, such as docker-linux-java17 or gpu-linux-cuda11, enabling precise agent selection in pipelines.

A significant gotcha with distributed builds is the assumption that the agent environment matches the controller environment. Build scripts that rely on specific file paths, environment variables, or installed tools may fail silently on agents with different configurations. Teams should use Docker-based or Kubernetes-based agents to ensure environment consistency, or use tool installers configured in Jenkins to automatically install required tools on agents. Another common mistake is running builds on the controller node itself, which can exhaust its resources and destabilize the entire Jenkins instance. Always set the number of executors on the controller to zero and force all builds to run on agents.

// Jenkinsfile for order-service with distributed agent configuration pipeline { // Do not assign a global agent; each stage picks its own agent none // Define pipeline-level environment variables environment { // Name of the service being built SERVICE = 'order-service' // Docker image registry URL REGISTRY = 'registry.company.com' } // Define the build stages with agent-per-stage strategy stages { // Build stage runs on a Java-capable agent stage('Build') { // Select an agent with the java17 label agent { label 'docker-linux-java17' } steps { // Check out the source code from version control checkout scm // Compile the application using Gradle wrapper sh './gradlew clean assemble' // Stash the built artifacts for use in later stages stash includes: 'build/libs/*.jar', name: 'app-jar' } } // Test stage runs on a separate agent to parallelize stage('Test') { // Use a Docker container as the agent for clean environment agent { docker { // Specify the Docker image with test dependencies image 'gradle:7.6-jdk17' // Mount the Gradle cache for faster builds args '-v /tmp/gradle-cache:/home/gradle/.gradle' } } steps { // Check out source again on this separate agent checkout scm // Run the full test suite inside the container sh './gradlew test integrationTest' // Publish JUnit test results to Jenkins junit 'build/test-results/**/*.xml' } } // Docker build stage runs on an agent with Docker socket stage('Docker Build') { // Select an agent that has Docker installed agent { label 'docker-builder' } steps { // Check out the Dockerfile and build context checkout scm // Retrieve the built JAR from the earlier stash unstash 'app-jar' // Build the Docker image tagged with the build number sh "docker build -t ${REGISTRY}/${SERVICE}:${BUILD_NUMBER} ." // Push the image to the company Docker registry sh "docker push ${REGISTRY}/${SERVICE}:${BUILD_NUMBER}" } } // Kubernetes deployment stage on an agent with kubectl stage('Deploy to Staging') { // Run on an agent with kubectl and helm installed agent { label 'kubernetes-deployer' } // Only deploy from the main branch when { branch 'main' } steps { // Update the Kubernetes deployment with new image tag sh "kubectl set image deployment/${SERVICE} ${SERVICE}=${REGISTRY}/${SERVICE}:${BUILD_NUMBER} -n staging" // Wait for the rollout to complete successfully sh "kubectl rollout status deployment/${SERVICE} -n staging --timeout=300s" } } } // Post-build actions run on the controller post { // Send notification on build failure failure { // Alert the team via Slack about the failed build slackSend channel: '#order-team', message: "${SERVICE} build ${BUILD_NUMBER} FAILED" } } }

How do you implement Jenkins agents/workers for distributed builds?

More Jenkins questions