How do Kubernetes Pod agents work in Jenkins for ephemeral build environments, and what should architects consider when designing pod templates, container definitions, and resource management?

Think of a pop-up kitchen at a food festival. Instead of maintaining permanent kitchens (static agents), you set up a temporary cooking station (Pod) with exactly the equipment needed for each dish, cook the dish, and tear down the station. The pod template is the equipment checklist — one station needs a grill and fryer (Docker and Maven containers), another needs an oven and mixer (Helm and Terraform containers). Each station exists only for the duration of one order (build) and leaves no mess behind.

Jenkins Kubernetes plugin extends the Jenkins agent model by creating Pods in a Kubernetes cluster for each build execution. The pod template defines the Pod specification: which containers to run, what volumes to mount, what resource requests and limits to set, and what node selectors or tolerations to apply. Every pod template automatically includes a JNLP container that connects back to the Jenkins controller via the Jenkins Remoting protocol. Additional containers defined in the template run alongside the JNLP container, and the pipeline uses the container('name') step to execute commands inside a specific container.

Internally, when a pipeline starts, the Kubernetes plugin sends a Pod creation request to the Kubernetes API server with the rendered pod template. The Pod schedules onto a node with sufficient resources, all containers start, and the JNLP container establishes a WebSocket or TCP connection to the Jenkins controller. The pipeline workspace is typically an emptyDir volume shared across all containers in the Pod, so files created in one container (like compiled artifacts from Maven) are accessible in another container (like Docker for image building). When the build completes or is aborted, the plugin deletes the Pod, and all workspace data is lost unless explicitly archived or pushed to external storage.

At production scale, pod template design directly impacts cluster cost, build speed, and reliability. Resource requests determine scheduling and cluster autoscaler behavior — under-requesting causes OOMKills and CPU throttling during builds, while over-requesting wastes cluster capacity and blocks other builds from scheduling. A common pattern is to define a base pod template with shared settings (tolerations for build-node taints, service accounts for registry access, resource defaults) and let teams inherit and extend it with their specific containers. Image pull time is a major latency factor — building on cold nodes with large images like eclipse-temurin:21 or node:20 can add 30-60 seconds. Pre-pulling images onto build nodes or using image caching solutions like Spegel reduces this.

The non-obvious gotcha is workspace persistence between pipeline stages. If a pipeline has multiple stages that each specify different pod templates (or if the Pod is rescheduled), the workspace from the previous stage is lost because it lived on the deleted Pod's emptyDir. Architects must either use a single pod template with all needed containers for the entire pipeline, use a PersistentVolumeClaim for the workspace (which adds complexity and storage cost), or explicitly stash/unstash artifacts between stages. Another gotcha is resource contention — if 50 builds trigger simultaneously and each Pod requests 2 CPU cores and 4 GB memory, the cluster needs 100 cores and 200 GB of memory available immediately or builds queue behind pending Pods.