How do you configure and scale GitLab Runners with the Kubernetes executor for large-scale CI/CD workloads?

Think of the Kubernetes executor like an on-demand temp agency for a construction company. Instead of maintaining a permanent workforce (static VMs) that sits idle between projects, the company calls the temp agency (Kubernetes) whenever a new project (CI job) comes in. The agency assembles a team (pod) with the exact skills needed (containers with specified images), the team completes the work, and then disbands. The company only pays for hours worked (pod resource usage), and the agency can scale to provide a hundred teams simultaneously during peak construction season.

The Kubernetes executor is one of several executor types supported by GitLab Runner. When configured, each CI/CD job is executed as a Kubernetes pod in the cluster. The pod typically contains at least three containers: a build container running the specified CI job image where the script commands execute, a helper container that handles git cloning, artifact uploading, and cache management, and optionally sidecar containers defined in the services keyword (like PostgreSQL or Redis for testing). The Runner itself runs as a long-lived pod or deployment in the cluster, polling GitLab for jobs. When it picks up a job, it creates a pod specification based on the config.toml settings and the job's requirements, submits it to the Kubernetes API, and monitors its execution. After the job completes, the pod is terminated and its resources are released.

Internally, the Kubernetes executor translates GitLab CI job definitions into Kubernetes pod specifications. The config.toml file allows extensive customization of the generated pods: namespace selection, service account assignment, resource requests and limits (CPU and memory), node affinity and tolerations for targeting specific node pools, volume mounts for persistent storage, pod annotations and labels for monitoring integration, and security context settings like running as non-root. The executor supports three pod cleanup strategies: on success (delete pods after successful jobs), always (delete all pods), and never (keep pods for debugging). For autoscaling, the cluster's node autoscaler works naturally with the Kubernetes executor: when many CI jobs create pods simultaneously, the pending pods trigger the autoscaler to add nodes, and when jobs complete and pods are deleted, the autoscaler removes unnecessary nodes.

In production, a company running 500 CI/CD pipelines per day across 100 microservices would deploy a multi-Runner Kubernetes architecture. The primary Runner deployment runs in a dedicated ci-runners namespace with service accounts scoped to only create pods in that namespace (preventing CI jobs from accessing production workloads). Node pools are configured with specific machine types: a general-purpose pool with e2-standard-4 instances for standard CI jobs, a high-memory pool with e2-highmem-8 instances for compilation-heavy jobs (targeted via node affinity and Runner tags), and a spot/preemptible pool for cost-sensitive jobs like nightly integration tests. Resource limits are set aggressively to prevent runaway jobs from starving the cluster: 2 CPU cores and 4GB RAM as defaults, with specific overrides for known resource-intensive jobs. Pod security policies enforce non-root execution, read-only root filesystems where possible, and disallow privilege escalation. The Runner is deployed via the official GitLab Runner Helm chart, which manages the Runner deployment, configures RBAC, and supports automatic Runner registration.

A critical gotcha is the Docker-in-Docker problem on Kubernetes. Many CI jobs need to build Docker images, which traditionally requires running a Docker daemon inside the CI container (privileged mode). On Kubernetes, this means granting the pod privileged security context, which is a significant security risk. The solution is using Kaniko, Buildah, or buildkit for rootless image builds that do not require a Docker daemon. Another common issue is pod scheduling delays: if the cluster needs to scale up nodes, jobs may wait minutes for a node to become available. Configure the cluster autoscaler's scale-up timeout and over-provision with pause pods (low-priority pods that are preempted when real jobs arrive) to reduce cold-start latency. Also, be aware of the DNS resolution delay in fresh pods; set the dnsPolicy to Default instead of ClusterFirst if CI jobs make heavy external API calls.