How do you enable Docker layer caching in Azure DevOps to speed up container builds?

Think of Docker layer caching like a construction site that stores pre-built wall sections in a warehouse between projects. Without caching, every project starts from raw lumber even if the wall design is identical to last week's build. With caching, the crew checks the warehouse first — if a matching wall section exists, they use it immediately and only build new sections for the parts that changed. Docker layer caching works the same way: unchanged layers from previous builds are reused, dramatically reducing build time for images where only application code changes while base layers (OS packages, dependencies) remain stable.

On Microsoft-hosted agents, the default Docker build experience has no caching because each pipeline run provisions a fresh virtual machine. The previous build's layers no longer exist, forcing Docker to rebuild every layer from scratch. This is particularly painful for large images with expensive package installation steps: a RUN apt-get install or RUN pip install layer that takes 3 minutes rebuilds every time even though the package list has not changed. For a payments-api with a multi-stage Dockerfile, uncached builds might take 8-10 minutes while cached builds complete in 1-2 minutes.

The Cache@2 task provides pipeline-level caching that persists between runs. You configure it to cache the Docker build cache directory (/var/lib/docker or BuildKit's cache) keyed on the Dockerfile content hash. When the Dockerfile has not changed, the cache restores the previous build layers. However, this approach has limitations: the cache download takes time proportional to its size, the Docker daemon's local layer store does not serialize cleanly for all scenarios, and cache invalidation happens at the Dockerfile hash level rather than the layer level.

Registry-based caching with BuildKit's --cache-to and --cache-from flags is the preferred approach for Microsoft-hosted agents. Instead of persisting the local cache between runs, you push cache metadata to a container registry after each build and pull it before the next build. BuildKit compares each layer's cache key against the registry cache and only rebuilds layers that changed. This approach is registry-native, works across different agents and even different pipelines, and uses inline cache or registry-type cache backends. The inline cache embeds cache metadata in the image itself, while the registry-type uses a separate cache manifest for finer-grained control.

Self-hosted agents provide the simplest caching story: since the Docker daemon persists between pipeline runs, layers from previous builds exist in the local layer store automatically. Docker's build cache works as designed without any additional configuration. The tradeoff is disk space management — without periodic cleanup, the Docker layer store grows unbounded. Teams must balance cache retention (keeping useful layers) with disk space (removing stale images). A common pattern is running docker system prune with age-based filters on a schedule rather than after every build.

The production gotcha is cache invalidation ordering in Dockerfiles. Docker's layer cache invalidates from the first changed layer downward: if layer 3 of 10 changes, layers 4-10 rebuild even if their content did not change. This means ordering matters critically: place infrequently-changing layers (OS packages, framework installation) before frequently-changing layers (application code copy). A common anti-pattern is COPY . . early in the Dockerfile, which invalidates all subsequent layers on every code change. Instead, copy dependency manifests first (package.json, requirements.txt), install dependencies, then copy application code. This maximizes cache hits for the expensive dependency installation layer.