What mTLS-specific signals should you alert on to catch a certificate rotation failure before it causes a mesh-wide outage rather than after?

This is like monitoring a fire alarm system not just for 'the building is currently on fire' but for the earlier, quieter signals — a smoke detector's battery reporting low, a sprinkler system's water pressure trending down — because reacting only when the alarm itself finally goes off means you've already lost the chance to prevent the fire, not just respond to it faster. mTLS operations have exactly this same shape: the catastrophic failure (mesh-wide handshake failures) is preceded by quieter, earlier signals — certificates approaching expiry, rotation propagation stalling partway through a fleet, or a rising share of traffic silently falling back to plaintext — that are far more useful to alert on precisely because they give you time to act before anything actually breaks.

This leading-versus-lagging distinction matters because mTLS failure modes are almost always binary and instantaneous once they hit — a certificate that's valid one moment and expired the next doesn't degrade gracefully, it simply starts rejecting every connection the instant the clock ticks past its expiry, which means a purely reactive alert ('handshakes are failing') fires at the exact moment the outage has already started, with essentially zero warning lead time. Good mTLS observability is specifically designed to catch the multi-hour or multi-day runway before that cliff, not just the cliff edge itself.

Step by step, the concrete signals to build dashboards and alerts around: first, certificate expiry countdown per workload, scraped directly from each sidecar's active certificate (via the mesh's own metrics or a periodic export), alerting when any certificate in the fleet drops under a safety threshold (say, 24 hours) without having already been rotated — this catches a stalled rotation before expiry, not after. Second, handshake failure rate, but segmented by workload identity and sidecar/proxy version rather than a single fleet-wide aggregate, since an aggregate metric can hide a 100% failure rate on one rollout wave inside an otherwise-healthy 99% success rate overall — segmentation is what turns a diluted, easy-to-miss signal into an obvious, immediately actionable one. Third, PERMISSIVE-mode plaintext fallback volume, for any mesh still in migration to STRICT mTLS — a rising percentage of connections falling back to plaintext isn't itself a rotation failure, but it's evidence that some population of workloads is failing mTLS handshakes silently and successfully hiding the fact behind the permissive fallback, which is precisely the scenario that lets a real problem go completely unnoticed until the policy is finally tightened to STRICT and everything relying on that plaintext fallback breaks at once.

At production scale, the most mature setup treats certificate rotation as a first-class deployment with its own health checks and rollback criteria, not a background cron job nobody watches — the propagation of a new CA root across every sidecar in the fleet should be tracked with the same rigor as a canary rollout, with an explicit gate ("do not proceed to issuing new leaf certificates until X% of sidecars confirm trust of the new root") rather than assuming propagation completed just because enough time has passed.

The gotcha: teams that only alert on the fleet-wide aggregate handshake success rate often set the threshold loose enough (say, alert below 99%) to avoid noise from normal transient blips, but that same threshold completely masks a rotation that has silently failed for one specific service representing far less than 1% of total mesh traffic — the aggregate metric stays green while that one service has been fully broken, sometimes for hours, because nobody segmented the alert by workload and the small blast radius kept it under the fleet-wide noise floor.