How does automatic mTLS certificate rotation work inside a service mesh sidecar, and what happens to in-flight requests on checkout-worker if the CA bundle rotates before every sidecar has reloaded it?

Think of a company that reissues every employee's security badge every 90 days, but instead of doing it all at once and having half the building unable to open any doors on changeover day, they quietly hand out new badges a week early while the old badges still work, and only deactivate the old badges once they're confident everyone has their new one. If they got this overlap wrong — deactivating old badges before everyone had picked up their new one — some employees would suddenly find every door in the building rejecting them, even though nothing about their actual identity or authorization changed. Service mesh certificate rotation is built around exactly this kind of deliberate overlap, because the alternative — a hard cutover — guarantees a window of self-inflicted outages every single rotation cycle.

Meshes issue certificates with intentionally short lifetimes (often 24 hours or less, compared to the months or years typical for a public-facing TLS certificate) specifically because short-lived certificates dramatically shrink the damage window if a private key is ever compromised — a leaked key that's only valid for a few hours is a far smaller problem than one valid for a year. But short lifetimes only work operationally if rotation is fully automated and invisible to whoever owns the workload; nobody wants to manually reissue certificates for hundreds of services every single day, so the mesh's control plane (Istio's istiod, for example) takes over the entire lifecycle.

Step by step: the control plane runs its own internal CA (or integrates with an external one like Vault or cert-manager). Each sidecar proxy periodically requests a new certificate before its current one expires, presenting proof of its workload identity (in Kubernetes, typically its service account token) to the control plane, which issues a fresh short-lived certificate signed by the current CA and pushes it down over a secure channel (Istio uses the Secret Discovery Service, SDS, for this). The sidecar swaps in the new certificate for future connections without dropping existing ones — in-flight connections continue using whichever certificate was active when the TLS session was established, since renegotiating mid-connection isn't necessary as long as the old certificate remains valid until its own expiry. When the CA *root* itself rotates (a much rarer, higher-stakes event than routine leaf certificate renewal), the mesh needs every sidecar to trust *both* the old and new root simultaneously for an overlap period, so that a sidecar that's already received its new leaf certificate (signed by the new root) can still be verified by a peer sidecar that hasn't yet updated its trust bundle to include the new root.

At production scale, what to monitor during any CA rotation is the handshake failure rate specifically segmented by sidecar version or rollout wave, since a spike isolated to sidecars that haven't yet received the updated trust bundle is expected and self-resolving, while a sustained or fleet-wide spike indicates the rotation's overlap window was too short or the rollout stalled partway through. Teams typically stage root CA rotations deliberately slowly — pushing the new root into every sidecar's trust bundle first, waiting for that to fully propagate and confirming via metrics, and only then beginning to issue new leaf certificates signed by the new root — precisely to avoid the scenario where new-root certificates start appearing before old-root trust has been fully distributed everywhere.

The gotcha: if checkout-worker's sidecar gets its new certificate before payments-api's sidecar has updated its trust bundle to include the new CA, the mTLS handshake between them fails — not because either certificate is invalid, but because payments-api's sidecar simply doesn't yet recognize the CA that signed checkout-worker's shiny new certificate. This produces a transient, seemingly random subset of failed requests clustered right around the rotation window that often gets misdiagnosed as a networking blip, when the actual cause is a rollout-ordering gap in an otherwise correctly-designed rotation process.