How would you automate SLO burn-rate alerting so that a single flaky, low-importance dependency doesn't page the entire payments-api on-call rotation at 3 AM?

This is like a hospital's alarm system paging the on-call surgeon every time a vending machine on a different floor malfunctions, just because both incidents technically happened somewhere in the building. The alarm system conflates 'something, somewhere, isn't working' with 'a patient's condition requires an immediate specialist,' and that conflation is exactly what erodes trust in the alarm over time — eventually the surgeon starts treating every page with skepticism, which is dangerous the one time it's a real emergency. SLO burn-rate alerting has to make the same distinction deliberately: not every internal failure deserves to page a human, only the ones that actually threaten the user-facing promise the SLO represents.

The design principle at the heart of this is symptom-based alerting versus cause-based alerting. An SLI should be defined around what the user actually experiences — did their request succeed within an acceptable time — not around whether every internal component behaved perfectly. If payments-api calls a non-critical fraud-scoring service with a fallback (skip scoring and proceed conservatively) when that dependency is slow or unavailable, then a flaky fraud-scoring dependency failing shouldn't show up as a payments-api SLI violation at all, because the user's actual request still succeeded — the fallback absorbed the failure exactly as designed. If the alerting doesn't distinguish this, every blip in that one non-critical dependency independently pages the entire on-call rotation for a problem that, by design, never reached a real user.

Step by step, building this correctly requires several deliberate choices: first, define the SLI's success criteria to reflect genuine end-to-end user outcome, explicitly excluding calls that were retried and succeeded, or that fell back gracefully and still returned a valid response to the user — the raw request either succeeded from the user's point of view or it didn't, and internal retries or fallbacks that worked should count as success, not failure. Second, for dependencies you've explicitly identified as non-critical (with a working fallback path), route their own failure metrics to a separate, lower-severity signal — track them, absolutely, since a fallback path being invoked constantly instead of rarely is itself useful information, but don't let that separate signal independently trigger the same page-worthy alert as the primary user-facing SLI. Third, where a dependency actually is critical (no fallback exists, and its failure genuinely does break the user-facing request), its failures should already show up correctly in the primary SLI's burn rate without needing a separate alert at all — the SLI-based approach is precisely designed so you don't need a hand-maintained list of "which dependencies matter," because a dependency that matters will already move the number that matters.

At production scale, this distinction is what prevents alert fatigue from a very common but often overlooked source: over-instrumentation of internal component health, layered on top of proper SLO-based alerting, effectively double-counts the same underlying problem through two different alerting paths — one appropriately symptom-based and page-worthy, one cause-based and noisy. Teams that carefully audit which of their alerts are cause-based versus symptom-based often discover the majority of their 3 AM pages trace back to internal component alerts that never actually represented real user impact, and retiring or downgrading those to tickets is frequently the single highest-leverage change available to reduce on-call burden without weakening actual incident detection at all.

The gotcha: teams that build fallback logic for a non-critical dependency but never update their alerting to match often keep an old cause-based alert ("page if fraud-scoring service error rate exceeds X%") that predates the fallback being built, so the fallback quietly and correctly protects users from ever seeing an impact, while the stale alert keeps paging on-call for a problem that, by design, no longer affects anyone — the fix isn't tuning the threshold on that old alert, it's recognizing the alert itself no longer represents anything a human needs to act on and retiring it in favor of trusting the primary, already-correct SLI.