How do you set up canary deployments using Istio traffic splitting?

Think of a canary deployment like testing a new recipe at a restaurant. Instead of swapping the entire menu overnight, you serve the new dish to 5% of diners first, watch their reactions closely, and only expand to more tables if the feedback is good. If customers complain, you pull the dish immediately and go back to the original. Istio gives you precise control to send exactly the percentage of customers you want to the new dish, and to pull it back in seconds if something goes wrong.

Canary deployment is a gradual release strategy where a new version of a service runs alongside the existing stable version, and a small slice of real production traffic is shifted to the new version over time. Istio implements this through VirtualService (for traffic weight distribution) and DestinationRule (for defining version subsets). The key advantage over Kubernetes-native rolling updates is precision: Kubernetes can only control traffic proportionally to the number of replicas (for example, 1 out of 10 pods gets roughly 10% of traffic), whereas Istio can route exactly 1% of traffic to a canary with a single pod while 99% continues hitting the stable version running on many pods. This separates deployment from release -- you can deploy the new version without exposing it to any users until you explicitly set the traffic split.

Under the hood, the Envoy sidecars implement weighted routing using a weighted cluster selection algorithm. When istiod pushes the VirtualService configuration to the sidecars, each Envoy proxy keeps a routing table with weight assignments for each destination subset. For every incoming request, Envoy generates a random number and uses the weights to decide which upstream cluster (subset) gets the request. This happens per-request, not per-connection, so traffic distribution stays accurate even with long-lived HTTP/2 or gRPC connections. The DestinationRule's subset labels get translated into Envoy's Endpoint Discovery Service (EDS) configuration, which maps each subset to specific pod IPs. When you update the VirtualService weights, istiod pushes the new config via xDS, and Envoy applies it within seconds without dropping any active connections thanks to its hot-restart mechanism.

In production, a typical canary workflow has several stages. First, deploy the new version (v2) alongside the existing version (v1) with the canary subset defined in the DestinationRule. Second, start with a 5% traffic split to the canary and watch key metrics: error rate (should stay below baseline), p99 latency (should not spike), and business metrics (conversion rates, transaction success). Third, use automated canary analysis tools like Flagger or Argo Rollouts integrated with Istio to automatically promote or roll back based on metric thresholds. Fourth, ramp up traffic in stages (5% to 20% to 50% to 100%) with monitoring at each step. For critical services, also set up header-based routing so internal teams can test the canary version before any production traffic reaches it.

A major gotcha is sticky sessions and stateful interactions. If a user starts a checkout flow on v1 and then gets routed to v2 for the next request, the session state may not be compatible. Use consistent hashing in the DestinationRule or session affinity headers to keep individual users on the same version throughout their session. Another common mistake is watching the wrong metrics. If you only check HTTP status codes, you might miss a canary that returns 200 OK but with wrong data. Always include business-level metrics in your canary analysis. Finally, remember that traffic splitting happens at the Envoy proxy level, so any traffic that bypasses the mesh (like direct pod-to-pod calls using IP addresses) will not follow the configured weights.