Requests from checkout-worker to payments-api start failing with 'certificate verify failed' right after a canary rollout — walk through how you'd determine whether this is an expired leaf cert, a SAN mismatch, or a CA trust gap introduced by the canary.

This scenario is like a store chain rolling out new employee ID badges to only the downtown branch first, and suddenly customers report the downtown branch's security gate rejecting staff badges — the natural instinct is to check the badges, but the more useful first question is what actually changed at that specific branch versus every other branch that's still working fine. A canary rollout, by definition, creates a temporary population of pods running different code (and potentially a different sidecar proxy version or configuration) alongside the stable population, so when a certificate error appears right at rollout time, the very first diagnostic move should be isolating whether it's the canary pods specifically that are failing, not the service as a whole, because that immediately tells you whether the rollout itself introduced the problem.

This matters because a canary is designed to expose exactly this class of issue safely — a small percentage of traffic hitting new code before it's fully rolled out, so that a rollout-introduced regression (like a sidecar image bump that changed default mTLS behavior, or a Helm values change that pointed the canary at a different CA bundle ConfigMap) only affects a fraction of requests instead of 100% of traffic. The failure you're describing is precisely the kind of issue canary deployments exist to catch before a full rollout makes it universal.

Step by step: first, identify which specific pods are producing the error — kubectl get pods -l version=canary and correlate error logs or metrics by pod name, not just service name, since payments-api as a whole might be a mix of stable and canary pods with only the canary subset actually failing. Second, once you've confirmed it's canary-specific, diff the canary pod's actual live certificate and trust bundle against a stable pod's using the same istioctl proxy-config secret inspection — this directly answers whether the canary is running a different sidecar image version (which might ship a different default CA bundle path), was deployed with a stale ConfigMap mount from before the last CA rotation, or has a namespace/service-account identity mismatch that resulted in the mesh issuing it a certificate with different SANs than expected. Third, check the canary's rollout manifest itself for anything that touches TLS-adjacent configuration — a common cause is a canary Helm values override that unintentionally pins an older sidecar image tag, which in turn ships an outdated CA trust bundle that predates a recent root rotation the stable pods already picked up.

At production scale, this exact pattern — a canary failing on mTLS while stable pods work fine — is usually not a certificate problem in the traditional sense at all, it's a deployment consistency problem: the canary was built from a slightly different base image, chart version, or config snapshot than the currently-running stable pods, and that drift happened to land on the one thing (trust bundle version) that breaks connectivity outright instead of failing more gracefully. Catching this requires comparing configuration, not just certificates, between the two pod populations.

The gotcha: engineers who only look at the canary pod's own certificate in isolation, without diffing it against a stable pod's, often conclude 'the certificate looks fine' — because it usually is a valid, correctly-signed certificate — and stay stuck, since the actual break is a trust *mismatch between populations*, not a defect in either certificate individually. The fix isn't reissuing a certificate at all; it's realizing the canary and stable pods are, transiently, running two different mTLS configurations that were never meant to diverge, and the resolution is aligning the canary's sidecar/config version with stable before promoting or rolling back the canary entirely.