How do you implement progressive delivery with Flux and Flagger, including canary deployments, A/B testing, and automated rollback for production services?

Flagger is a progressive delivery operator that integrates tightly with Flux to add automated canary analysis, A/B testing, and blue-green deployments to the GitOps workflow. While Flux handles the reconciliation of desired state from Git to the cluster, Flagger intercepts the deployment process and adds a controlled rollout phase where the new version is gradually exposed to production traffic. If the new version meets predefined service level objectives based on real metrics, the rollout proceeds to completion. If metrics degrade, Flagger automatically rolls back without human intervention. This transforms Flux from a simple apply engine into a sophisticated deployment platform that can safely release changes to high-traffic production services.

Flagger works by creating a shadow deployment alongside the primary deployment managed by Flux. When Flux updates the target deployment's container image or configuration, Flagger detects the change and begins a canary analysis. It creates a canary deployment with the new version, configures the service mesh or ingress controller to route a small percentage of traffic to the canary, and starts querying metric providers to evaluate the canary's health. The traffic percentage increases in configurable steps, with each step requiring the metrics to remain within acceptable thresholds for a defined interval. For a service like payments-api handling thousands of transactions per minute, this means the new version processes only a small fraction of real traffic initially, limiting blast radius.

The Canary custom resource is where all progressive delivery configuration lives. It references the target deployment, defines the analysis interval and threshold, specifies metric queries for evaluation, and configures webhooks for integration with external systems. Metrics can come from Prometheus, Datadog, CloudWatch, New Relic, or custom metric providers. A typical configuration for a payments service would check the HTTP 5xx error rate, P99 latency, and business metrics like transaction success rate. The analysis section defines how many iterations must pass and the maximum acceptable failure count before Flagger declares the canary healthy or initiates a rollback.

A/B testing with Flagger adds header-based or cookie-based routing to the progressive delivery workflow. Instead of shifting a percentage of all traffic, A/B testing routes specific users or requests to the canary based on HTTP headers like x-canary: true or cookies set by feature flag systems. This is particularly useful for testing user-facing changes where you want specific customer segments to experience the new version before broader rollout. Flagger configures the service mesh or ingress to perform this routing and still evaluates metrics to ensure the canary performs well for the targeted users.

Operating Flagger in production requires robust metric pipelines and well-defined SLOs. If the metrics pipeline has gaps or delays, Flagger may advance the canary based on incomplete data. Teams should configure alerting for Flagger events including canary initialization, weight progression, and rollback triggers. Webhooks can integrate with Slack for notifications, with load testing tools like Flagger's loadtester for synthetic traffic during canary analysis, and with external approval systems for gated promotions. Multi-cluster deployments need Flagger installed in each cluster with cluster-specific metric queries, as latency baselines and error budgets may differ between regions. The interaction between Flux and Flagger is important to understand: Flux owns the deployment spec and updates it from Git, while Flagger manages the rollout lifecycle. If Flux force-applies during an active canary, it can disrupt the progressive delivery process, so reconciliation intervals should be longer than the maximum canary analysis duration.