Your CI pipeline takes 25 minutes and developers are complaining. The pipeline runs lint, build, unit tests, integration tests, SAST scan, Docker build, and deployment. How do you optimize it using DAG pipelines, caching, artifacts, and parallel execution?
Quick Answer
Use needs keyword for DAG execution (skip stage ordering), parallel keyword for splitting test suites, distributed caching for dependencies, Docker layer caching with BuildKit, and rules to skip unnecessary jobs. Target: reduce from 25 min to under 8 min.
Detailed Answer
Identifying Bottlenecks
Before optimizing, analyze the pipeline using GitLab's pipeline analytics (CI/CD > Pipelines > Charts) and job duration data. Common bottlenecks: sequential stage execution when jobs are independent, downloading dependencies every run, full test suite running serially, and Docker builds without layer caching. Enable CI_DEBUG_TRACE on slow jobs to identify I/O vs CPU bottlenecks.
DAG Execution with needs Keyword
By default, GitLab runs all jobs in a stage only after the previous stage completes. The needs keyword breaks this constraint—a job starts immediately when its dependencies finish, regardless of stage. Example: Docker build can start as soon as unit tests pass, without waiting for the SAST scan in the same stage. This alone can save 30-50% of pipeline time by overlapping independent work.
Parallel Test Execution
The parallel keyword splits a job into N instances. Combined with test splitting tools (like split-tests for Go or jest --shard for JavaScript), you distribute the test suite across parallel runners. For a 15-minute test suite, parallel: 5 reduces it to ~3 minutes plus overhead. GitLab provides $CI_NODE_INDEX and $CI_NODE_TOTAL variables for custom splitting logic.
Caching Strategy
Cache dependency directories (node_modules, vendor, .gradle) with keys based on lock files. Use cache:policy:pull for jobs that only read cache and push for jobs that update it. Distribute cache via S3/GCS for multi-runner setups—local filesystem caching doesn't work when jobs land on different runners. Set cache:when: always to update cache even on job failure, preventing stale caches.
Docker Build Optimization
Use BuildKit with --cache-from pointing to the registry for layer caching. Multi-stage builds with strategic layer ordering (dependencies before source code) maximize cache hits. For projects with complex builds, use kaniko or buildx with registry-backed cache. Enable Docker BuildKit's inline cache export: --build-arg BUILDKIT_INLINE_CACHE=1. This single optimization often saves 5-10 minutes on Docker-heavy pipelines.