Design a CI/CD pipeline for a monorepo with 200 microservices. How do you handle selective builds, dependency ordering, and deployment orchestration?
Quick Answer
Use change detection (git diff) for selective builds, build a dependency DAG for ordering, parallelize independent builds, and use progressive delivery with automated canary analysis for deployments.
Detailed Answer
Selective Builds (Only Build What Changed)
1. Change detection: Compare git diff HEAD~1 --name-only against a service-to-path mapping 2. Dependency graph: If shared-lib changes, rebuild all services that depend on it 3. Path-based triggers: GitHub Actions paths filter or Jenkins changeset trigger 4. Build graph tool: Use Nx, Bazel, or Turborepo for intelligent caching and dependency-aware builds
Pipeline Architecture
`` PR → Lint/Format → Unit Tests → Build → Integration Tests → Security Scan ↓ Merge to main → Build → Push Image → Deploy Canary → Canary Analysis → Full Rollout ``
Parallelization
- Independent services build in parallel (fan-out) - Shared libraries build first (topological sort of dependency DAG) - Use build caching (Docker layer cache, npm/Maven cache) to speed up builds - Matrix builds for multi-arch (amd64/arm64)
Deployment Orchestration
- ArgoCD ApplicationSets: Auto-generate ArgoCD Applications per service - Wave-based rollout: Deploy infrastructure changes first, then shared services, then application services - Canary analysis: Automated metrics comparison (Argo Rollouts + Prometheus) before full promotion - Rollback: Automated rollback if error rate or latency degrades during canary
Key Challenges at Scale
- Build queue management: 200 services × multiple PRs = queue pressure. Use auto-scaling CI runners. - Flaky tests: Track test reliability metrics, quarantine flaky tests automatically - Artifact management: ECR/Artifactory with lifecycle policies to control storage costs - Secret management: CI secrets in Vault, rotated automatically, never in pipeline YAML
Code Example
# GitHub Actions: Selective build with path filters
name: CI
on:
push:
branches: [main]
pull_request:
jobs:
detect-changes:
runs-on: ubuntu-latest
outputs:
services: ${{ steps.changes.outputs.services }}
steps:
- uses: actions/checkout@v4
with: { fetch-depth: 0 }
- id: changes
run: |
changed=$(git diff --name-only ${{ github.event.before }} HEAD | \
grep -oP 'services/\K[^/]+' | sort -u | jq -R -s -c 'split("\n") | map(select(. != ""))')
echo "services=$changed" >> $GITHUB_OUTPUT
build:
needs: detect-changes
strategy:
matrix:
service: ${{ fromJson(needs.detect-changes.outputs.services) }}
steps:
- run: echo "Building ${{ matrix.service }}"Interview Tip
Monorepo CI/CD at scale is a common staff+ question. Key signals: change detection with dependency awareness, build caching, parallel execution, and progressive delivery. Mention specific tools (Nx/Bazel for builds, ArgoCD for deployment, Argo Rollouts for canary).