intermediateGitLab CI/CD

Your GitLab CI pipeline has a complex dependency graph with 40+ jobs. Some jobs are taking longer than expected due to incorrect artifact dependencies and sequential execution where parallelism is possible. How do you diagnose and optimize pipeline execution using DAG visualization, needs keyword, and resource classes?

⚡

Quick Answer

Use GitLab's pipeline graph visualization to identify the critical path, convert stage-based execution to DAG with needs keyword, eliminate unnecessary artifact downloads with dependencies keyword, use resource_group for deployment serialization, and implement parallel with test splitting.

Detailed Answer

Diagnosing Pipeline Bottlenecks

GitLab's pipeline visualization shows job dependencies and execution timeline. The critical path is the longest sequential chain of jobs—this is your pipeline's minimum possible duration. Jobs outside the critical path can be optimized but won't reduce total time. Use CI/CD Analytics to see median durations over time and identify regressions. The job log's Preparing executor time reveals runner acquisition delays separate from actual execution time.

DAG vs Stage-Based Execution

Default GitLab behavior: all jobs in stage N must complete before any job in stage N+1 starts. This is wasteful when jobs have no data dependencies. The needs keyword creates a DAG: a job starts as soon as its declared dependencies complete, regardless of stage. Example: SAST scanning (no build dependency) can start immediately, while deploy (depends on build + test) waits only for those specific jobs. Stage names become purely organizational.

Artifact Optimization

By default, all jobs in a stage download all artifacts from the previous stage—even if they don't need them. dependencies: [] prevents downloading any artifacts. dependencies: [job-name] downloads only from specified jobs. For DAG pipelines, needs implicitly sets dependencies. Large artifacts (Docker images, compiled binaries) add significant transfer time; use artifacts:expire_in aggressively and consider registry storage for images instead of CI artifacts.

Resource Groups for Deployment

resource_group ensures only one job with that group name runs at a time—essential for deployments where two simultaneous deploys would conflict. Unlike stages, resource_group works across pipelines: if pipeline A is deploying, pipeline B's deploy job queues until A finishes. This prevents race conditions without blocking the entire pipeline.

Pipeline Efficiency Metrics

Track: pipeline duration (wall clock), total job time (sum of all job durations), and efficiency ratio (total time / duration / concurrent jobs). A 30-minute pipeline with 60 minutes of total job time running 10 concurrent jobs has poor efficiency—most jobs are waiting. Aim for the critical path to be less than 20% of total job time, indicating good parallelization.

# Before optimization: 40-minute sequential pipeline # After optimization: 12-minute DAG pipeline stages: - validate # 3 min total - build # 5 min total - test # 15 min total - security # 8 min total - package # 4 min total - deploy # 3 min total # Jobs with explicit DAG dependencies lint: stage: validate needs: [] # Start immediately, no dependencies script: golangci-lint run schema-validate: stage: validate needs: [] # Parallel with lint script: openapi-generator validate -i api/openapi.yaml build-binary: stage: build needs: ["lint"] # Only needs lint, not schema-validate script: go build -o app ./cmd/server artifacts: paths: [app] expire_in: 2 hours build-frontend: stage: build needs: [] # Frontend doesn't depend on Go lint script: cd frontend && npm ci && npm run build artifacts: paths: [frontend/dist/] expire_in: 2 hours # Parallel test execution unit-test: stage: test needs: ["build-binary"] # Starts as soon as binary builds parallel: 4 script: - PACKAGES=$(go list ./... | split-tests -n $CI_NODE_TOTAL -i $CI_NODE_INDEX) - go test -race $PACKAGES artifacts: reports: junit: report-$CI_NODE_INDEX.xml integration-test: stage: test needs: ["build-binary"] # Parallel with unit tests! services: - postgres:16 script: go test -tags=integration ./tests/... frontend-test: stage: test needs: ["build-frontend"] # Only depends on frontend build dependencies: ["build-frontend"] # Only download frontend artifacts script: cd frontend && npm test # Security scans run parallel to tests (no build dependency) sast: stage: security needs: [] # Start at pipeline begin! include: - template: Security/SAST.gitlab-ci.yml dependency-scan: stage: security needs: [] # Parallel with everything include: - template: Security/Dependency-Scanning.gitlab-ci.yml # Docker build only needs binary + frontend docker-build: stage: package needs: ["build-binary", "build-frontend"] dependencies: ["build-binary", "build-frontend"] # Only these artifacts script: - docker buildx build --push --tag $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA . # Deploy with resource_group prevents concurrent deploys deploy-staging: stage: deploy needs: ["docker-build", "unit-test", "integration-test"] dependencies: [] # No artifacts needed for deploy resource_group: staging-deploy # Serialize across pipelines environment: name: staging script: - kubectl set image deploy/app app=$CI_REGISTRY_IMAGE:$CI_COMMIT_SHA # Critical path: lint(1m) -> build(2m) -> unit-test(4m) -> docker-build(3m) -> deploy(2m) = 12 min # vs sequential: 3+5+15+8+4+3 = 38 min # Pipeline efficiency check (run as scheduled job) pipeline-metrics: rules: - if: $CI_PIPELINE_SOURCE == 'schedule' script: - | curl -s --header "PRIVATE-TOKEN: $TOKEN" \ "https://gitlab.com/api/v4/projects/$CI_PROJECT_ID/pipelines?per_page=20&status=success" | \ jq '[.[] | {id, duration: .duration, created: .created_at}] | {avg_duration: (map(.duration) | add / length / 60 | round), count: length}'

More GitLab CI/CD questions