You need to implement multi-project pipelines where a platform library change triggers downstream service pipelines for testing. How do you set up cross-project triggers, pass variables, and handle the fan-out/fan-in pattern in GitLab?
Quick Answer
Use trigger keyword with project path to spawn downstream pipelines, pass variables via the trigger block, use strategy:depend to wait for downstream completion, and implement bridge jobs for fan-in. Use API triggers for complex orchestration patterns.
Detailed Answer
Multi-Project Pipeline Architecture
In microservice architectures, a shared library change can break downstream services. Multi-project pipelines let you trigger pipelines in other projects when a library is updated. The upstream pipeline creates bridge jobs that spawn downstream pipelines, optionally waiting for their completion before proceeding. This ensures integration compatibility is validated before the library is published.
Trigger Mechanisms
GitLab provides two trigger approaches: the trigger keyword (declarative, in .gitlab-ci.yml) and the Pipeline API (imperative, for complex orchestration). The trigger keyword is simpler—you specify the downstream project, branch, and variables. The downstream pipeline runs with its own .gitlab-ci.yml but receives the passed variables. strategy: depend makes the bridge job wait for the downstream pipeline to complete and mirrors its status.
Variable Passing and Artifact Sharing
Pass variables from upstream to downstream using the variables block within trigger. For artifact sharing across projects, use the needs:project keyword—a downstream job can fetch artifacts from a specific job in the upstream pipeline. This enables patterns like: library builds a package, downstream services download and test against it. Use CI_JOB_TOKEN for authentication between projects in the same group.
Fan-Out/Fan-In Pattern
Fan-out: a single upstream job triggers multiple downstream project pipelines in parallel. Each bridge job is independent and starts simultaneously. Fan-in: after all downstream pipelines complete (using strategy:depend on each bridge job), a final upstream job runs to aggregate results or publish the library. This pattern validates that a library change doesn't break any consumer before release.
Production Considerations
Limit downstream triggers to avoid CI resource exhaustion—a library used by 50 services shouldn't trigger 50 full pipelines on every commit. Use lightweight smoke test pipelines in downstream projects specifically for cross-project validation. Implement circuit breakers: if a downstream project's main branch is already broken, skip triggering it. Store pipeline status in a shared state (Redis or GitLab API) for complex orchestration beyond what YAML can express.