SRE Metrics

Burn rate is like judging whether a car is going to run out of gas before reaching its destination — knowing the fuel gauge reads 'half full' tells you almost nothing on its own; what actually matters is the rate of consumption relative to the remaining distance. A car burning fuel at a rate that would empty the tank in 500 miles is perfectly fine for a 200-mile trip and alarming for a 1,000-mile one. An SLO's error budget is the fuel tank, the 30-day (or whatever) SLO window is the trip distance, and burn rate is precisely that consumption-rate-relative-to-remaining-distance calculation, expressed as a multiple of the 'sustainable' rate that would use exactly 100% of the budget by the end of the window and no more.

This multiple-of-sustainable-rate framing was designed specifically to solve the problem of a single fixed error-rate threshold being either too noisy or too slow depending on how long the bad behavior persists. A fixed alert like 'page if error rate exceeds 1%' either fires on every brief, self-resolving blip (paging for something that would have consumed a negligible fraction of the actual monthly budget) or, if you raise the threshold to avoid that noise, fails to catch a sustained, lower-level degradation that quietly exhausts the entire monthly budget over several days without ever crossing the naive fixed threshold. Burn rate solves this by making the alert's severity a function of both how bad the error rate is and how long it's projected to last, expressed relative to the budget itself rather than an arbitrary absolute number.

Step by step, the calculation multiplies together two dimensions on purpose: burn rate equals the observed error rate divided by the budget's allowed error rate (1 minus the SLO target), evaluated over a specific lookback window. A 99.9% SLO allows a 0.1% error rate as its sustainable baseline; observing a 1.44% error rate over a short window is a burn rate of 14.4x — at that pace, the entire 30-day budget would be gone in about 30/14.4 ≈ 2 days, which is genuinely urgent and worth an immediate page. The same 1.44% error rate observed as an average over a much longer 6-hour window, if it's actually a brief spike that's already resolved, produces a much smaller *sustained* burn-rate reading because the long window dilutes a short spike down toward the noise floor — this is precisely why Google's SRE workbook recommends alerting on *multiple* window-and-threshold pairs simultaneously (commonly a fast 1-hour/5-minute pair for immediate paging and a slower 6-hour/30-minute pair for ticket-level investigation) rather than picking just one window size, since a single window size is always a trade-off between catching short severe spikes and catching long slow leaks.

At production scale, teams typically implement a two-tier (sometimes four-tier) burn-rate alerting scheme: a high-multiplier, short-window alert (e.g., 14.4x sustained for 2 minutes) pages immediately because it implies the budget could be fully gone within hours if unaddressed, while a lower-multiplier, longer-window alert (e.g., 3x sustained for an hour) creates a ticket for investigation during business hours because at that pace, the budget still has days of runway left even if the trend continues unaddressed. The engineering value of this design is that it makes the page-versus-ticket decision an automatic, calculated output rather than something an on-call engineer has to mentally estimate under pressure during an actual incident.

The gotcha: teams that implement burn-rate alerting with only a single window size inevitably end up either paging on harmless short blips (window too short) or missing a real, slow-building budget-exhausting trend until it's too late to fix gracefully (window too long) — the fix genuinely requires at least two window-and-multiplier pairs evaluated in parallel, and skipping this dual-window design is one of the most common mistakes teams make when they implement error budgets for the first time, often discovering the gap only after either alert fatigue or a missed slow-burn incident forces them to revisit the design.

A sudden 60-percentage-point budget drop in six hours is like a household's bank balance dropping sharply in a single afternoon — the sensible first move isn't reviewing every transaction from the past month, it's pulling up exactly what happened in that afternoon's window, because a change that dramatic almost never comes from gradual, ordinary spending, it comes from one identifiable event: a large purchase, a fraud charge, a mistaken transfer. Error budgets behave the same way — organic, gradual drift rarely produces a cliff this steep this fast, so the investigation should start by asking 'what changed in this exact six-hour window,' not by re-examining broad service health from scratch.

This correlation-first approach exists because budget burn, by construction, is just an aggregate of a huge number of individual failed or slow requests, and that aggregate number alone tells you nothing about causation — it's a symptom-level metric, not a diagnostic one. The entire reason to track deploy history, dependency health, and configuration changes alongside SLO dashboards is so that when the aggregate number moves sharply, you have an immediate, pre-correlated list of candidate causes to check, rather than starting a root-cause investigation completely from scratch under time pressure.

Step by step: first, pull the deploy timeline for checkout-worker itself and every service it directly depends on, and overlay it against the exact start time of the burn — if a deploy landed in or near that six-hour window, that's the leading suspect and should be checked first, since deploys are statistically the single most common trigger of sudden reliability regressions. Second, if no relevant deploy lines up, check the health and latency of checkout-worker's direct dependencies (databases, downstream APIs, message queues) over that same window, since an upstream degradation can burn a *downstream* service's budget even when the downstream service's own code never changed at all. Third, break down the failing requests themselves by endpoint and status code — a burn concentrated on one specific endpoint or a single error code (say, every failure is a 503 from one particular downstream call) narrows the search dramatically faster than treating the whole service as uniformly degraded. Fourth, check whether the burn is still actively ongoing at the moment of investigation or whether it was a discrete, already-resolved incident (a bad deploy that was already rolled back, a transient dependency blip that's already recovered) — this distinction is the actual determinant of what happens next.

At the organizational level, this is exactly where the pre-agreed freeze policy gets tested for real: if the burn is over and resolved, the appropriate response is a postmortem and, if the remaining budget for the window is now thin, heightened caution on further risky changes — but a full feature freeze over an already-resolved incident is usually disproportionate. If the burn is still active and ongoing, that's precisely the scenario the freeze policy was designed for: continuing to ship new changes into an environment that's already actively burning budget compounds risk on top of an unresolved problem, and the freeze exists to stop adding new variables until the existing one is under control.

The gotcha: teams that treat 'trigger the freeze' as an automatic, mechanical response the instant budget drops below a threshold — without first distinguishing an already-resolved historical burn from an actively ongoing one — end up freezing feature work over incidents that are already fixed, which trains the organization to see the freeze policy as arbitrary and punitive rather than a genuinely protective mechanism, and erodes the willingness of product stakeholders to honor the policy the next time it's actually needed for a real, ongoing problem.

Most teams instrument their services obsessively — latency, error rate, saturation, traffic — but treat the health of the humans operating those services as something you just informally sense from morale or notice only after someone quits. That's like a factory meticulously monitoring every machine on the line while never checking whether the workers running those machines are being pushed past a sustainable pace — the machines can look perfectly healthy on every dashboard right up until the workers operating them start walking out, at which point it's far too late to have caught the problem early. On-call sustainability deserves the same instrumentation discipline as the services it protects, precisely because human burnout, like service degradation, has measurable leading indicators long before it becomes a visible, painful lagging one.

This reframing — treating the on-call process itself as a system with SLIs, not just a duty roster — exists because burnout is otherwise almost always caught too late. Exit interviews and engagement surveys are lagging indicators; by the time someone reports burnout in a survey or, worse, resigns, the underlying operational conditions that caused it have usually been present and worsening for months. A platform team that only reacts to those late signals is running on-call reactively in exactly the same way a service that only alerts on outright downtime, rather than leading indicators of degradation, is running reliability reactively.

Step by step, the metrics worth building dashboards around: toil percentage — the fraction of on-call hours spent on manual, repetitive interventions that could in principle be automated, tracked weekly per person and per team, with a rising trend flagged well before it crosses an unsustainable threshold (commonly cited around 50%). Pages per shift — both the raw count and its trend over time, since a rotation that used to average one page per shift creeping toward four or five over a few months is a measurable early warning even if no single week looks alarming in isolation. After-hours and overnight page rate specifically, separated from business-hours pages, because a page at 3 AM has a categorically different human cost than the same page at 2 PM, and blending them into one aggregate metric hides exactly the dimension that correlates most directly with burnout and sleep disruption. Mean time to acknowledge and its trend — a rotation where acknowledgment times are quietly creeping up over weeks is often an early, indirect signal that on-call engineers are increasingly reluctant to engage quickly, frequently because they've learned from experience that pages are usually low-value noise rather than genuine emergencies.

At organizational scale, the most mature teams review these on-call health metrics in the same recurring operational review cadence as service-level reliability metrics — not as a one-off survey exercise, but as a standing agenda item with an owner and a trend line, exactly like an SLO dashboard. This normalizes discussing on-call sustainability as an ongoing engineering concern with real data behind it, rather than an occasional, hard-to-quantify cultural conversation that only comes up after morale has already visibly deteriorated.

The gotcha: teams that track these metrics in aggregate across an entire large team can miss a severe, concentrated burnout risk sitting on just one or two individuals — if one person is disproportionately picking up extra shifts, covering for a teammate on leave, or handling most of the after-hours pages because they happen to respond fastest, a team-wide average of 'reasonable' pages per shift can look completely healthy while that one specific person is quietly approaching burnout, invisible until the aggregate view is broken down per individual rather than only viewed at the team level.

This is like a hospital's alarm system paging the on-call surgeon every time a vending machine on a different floor malfunctions, just because both incidents technically happened somewhere in the building. The alarm system conflates 'something, somewhere, isn't working' with 'a patient's condition requires an immediate specialist,' and that conflation is exactly what erodes trust in the alarm over time — eventually the surgeon starts treating every page with skepticism, which is dangerous the one time it's a real emergency. SLO burn-rate alerting has to make the same distinction deliberately: not every internal failure deserves to page a human, only the ones that actually threaten the user-facing promise the SLO represents.

The design principle at the heart of this is symptom-based alerting versus cause-based alerting. An SLI should be defined around what the user actually experiences — did their request succeed within an acceptable time — not around whether every internal component behaved perfectly. If payments-api calls a non-critical fraud-scoring service with a fallback (skip scoring and proceed conservatively) when that dependency is slow or unavailable, then a flaky fraud-scoring dependency failing shouldn't show up as a payments-api SLI violation at all, because the user's actual request still succeeded — the fallback absorbed the failure exactly as designed. If the alerting doesn't distinguish this, every blip in that one non-critical dependency independently pages the entire on-call rotation for a problem that, by design, never reached a real user.

Step by step, building this correctly requires several deliberate choices: first, define the SLI's success criteria to reflect genuine end-to-end user outcome, explicitly excluding calls that were retried and succeeded, or that fell back gracefully and still returned a valid response to the user — the raw request either succeeded from the user's point of view or it didn't, and internal retries or fallbacks that worked should count as success, not failure. Second, for dependencies you've explicitly identified as non-critical (with a working fallback path), route their own failure metrics to a separate, lower-severity signal — track them, absolutely, since a fallback path being invoked constantly instead of rarely is itself useful information, but don't let that separate signal independently trigger the same page-worthy alert as the primary user-facing SLI. Third, where a dependency actually is critical (no fallback exists, and its failure genuinely does break the user-facing request), its failures should already show up correctly in the primary SLI's burn rate without needing a separate alert at all — the SLI-based approach is precisely designed so you don't need a hand-maintained list of "which dependencies matter," because a dependency that matters will already move the number that matters.

At production scale, this distinction is what prevents alert fatigue from a very common but often overlooked source: over-instrumentation of internal component health, layered on top of proper SLO-based alerting, effectively double-counts the same underlying problem through two different alerting paths — one appropriately symptom-based and page-worthy, one cause-based and noisy. Teams that carefully audit which of their alerts are cause-based versus symptom-based often discover the majority of their 3 AM pages trace back to internal component alerts that never actually represented real user impact, and retiring or downgrading those to tickets is frequently the single highest-leverage change available to reduce on-call burden without weakening actual incident detection at all.

The gotcha: teams that build fallback logic for a non-critical dependency but never update their alerting to match often keep an old cause-based alert ("page if fraud-scoring service error rate exceeds X%") that predates the fallback being built, so the fallback quietly and correctly protects users from ever seeing an impact, while the stale alert keeps paging on-call for a problem that, by design, no longer affects anyone — the fix isn't tuning the threshold on that old alert, it's recognizing the alert itself no longer represents anything a human needs to act on and retiring it in favor of trusting the primary, already-correct SLI.

An error budget is like a household's monthly discretionary spending limit on top of fixed bills — you're not expected to spend zero on entertainment, you're allowed a specific, agreed amount, and the moment you blow through that amount, the sensible response is not to keep spending as normal and hope it works out, it's to stop discretionary spending until next month's budget resets. SLOs and error budgets formalize exactly this idea for reliability: perfection (100% uptime, zero errors) is neither achievable nor actually necessary for most services, so instead of chasing an impossible target, teams agree in advance on how much unreliability is acceptable, and then treat that allowance as a real, trackable budget rather than a vague aspiration.

This framework exists because, without it, reliability work and feature work compete for the same engineering time with no objective tiebreaker — every team believes their velocity matters and every team believes reliability matters, and without a quantified, pre-agreed threshold, that tension gets resolved by whoever argues loudest in a planning meeting rather than by data. Google's SRE model introduced error budgets specifically to convert 'how reliable is reliable enough' from a subjective argument into an objective, numeric answer that both the reliability-focused and feature-focused sides of an organization agree to up front, before there's a live incident to argue about.

Step by step: first, define the SLI — pick the specific, measurable signal that represents user-facing quality, commonly success rate (percentage of requests that don't return a 5xx) or latency (percentage of requests under some threshold, like p99 under 200ms). Second, set the SLO as a target percentage of that SLI over a rolling window, commonly 30 days — 99.9% is a common target implying roughly 43 minutes of acceptable downtime or SLI-violating time per month. Third, the error budget is simply the inverse: 1 minus 99.9% equals 0.1%, which converts into that same ~43 minutes over 30 days, or however many failed requests that percentage represents given your traffic volume. Fourth, track budget consumption continuously — not just after the fact — using recording rules that calculate a rolling burn rate (how fast you're consuming the budget relative to a sustainable pace), so the team knows in real time whether they're on track to exhaust the budget early or comfortably coast to the end of the window with room to spare.

At production scale, the actual mechanics matter as much as the definition: teams implement this with Prometheus recording rules that continuously calculate the SLI over rolling windows, feed that into a burn-rate calculation, and alert on two distinct thresholds — a fast-burn alert (consuming budget so quickly that at the current rate you'd exhaust it in hours, demanding immediate paging) and a slow-burn alert (a steadier, less urgent trend that would exhaust the budget in days, worth investigating but not necessarily waking anyone up at 3 AM). What actually happens when the budget hits zero is the part organizations most often get wrong in practice: the 'freeze new features until reliability improves' consequence only has teeth if it was agreed to by both engineering leadership and product stakeholders *before* the budget ran out — negotiating it in the moment, mid-crisis, means whoever has the most organizational leverage simply overrides it, and the entire framework collapses back into the same subjective argument it was designed to eliminate.

The gotcha: teams that only look at the error budget number at the end of the 30-day window, rather than tracking burn rate continuously, discover the budget is already gone with two weeks still left in the window and have no way to react proactively — by definition, a lagging, retrospective view of budget consumption can only tell you the crisis already happened, never that it's *about* to happen, which is exactly why burn-rate alerting (not just a raw budget-remaining dashboard) is the difference between an SRE practice that actually prevents outages and one that just narrates them after the fact.

Imagine a fire department that gets called out constantly for burst water pipes, not actual fires — adding more firefighters to the roster does reduce how often any single firefighter gets called, but it doesn't address the real problem, which is that most of the calls shouldn't require a firefighter at all. On-call burnout works exactly the same way: the actual cause is almost never 'too few people in the rotation,' it's alerts that page a human for things that don't need urgent human judgment — noisy, low-value pages that erode trust in the alerting system and train people to view every page with dread rather than as a meaningful signal.

This is why the on-call design problem should be tackled as an alert-quality problem first and a staffing-math problem second. Google's SRE practice frames this explicitly through the concept of toil — repetitive, manual, automatable operational work that scales linearly with service growth rather than getting cheaper over time — and recommends capping toil at roughly 50% of an SRE's time specifically because toil that exceeds that threshold means the team is spending more time reacting to the same categories of problems than it is spending building the automation that would prevent those problems from recurring, a trap that gets structurally worse, not better, the longer it's tolerated.

Step by step, a well-designed rotation layers several independent safeguards: first, alert tiering — only genuinely customer-impacting symptoms (elevated error rate, breached latency SLO, full service unavailability) should page a human immediately; internal, non-urgent signals (a single retryable failed job, a metric drifting but still within a safe range) should generate a ticket or a dashboard entry, not a 2 AM page. Second, a primary-plus-secondary structure means a single missed or delayed acknowledgment doesn't leave an incident unattended — the secondary is a deliberate redundancy, not just a backup name on a list nobody expects to actually page. Third, hard caps on pages per shift (a common target is fewer than two genuine pages per shift) act as a forcing function: if a service consistently exceeds that cap, that's treated as a signal the alerting or the service itself needs engineering attention, not just something to individually tolerate. Fourth, blameless postmortems — analyzing what happened and why without assigning individual fault — exist because on-call engineers who fear blame for an incident start hiding problems or delaying escalation, which directly undermines the psychological safety needed for people to sustainably volunteer for high-stakes rotations at all.

At organizational scale, teams that track toil explicitly (logging roughly how much on-call time each week goes to manual, repeatable interventions versus genuine incident response or engineering work) get an early, objective warning sign long before individual burnout becomes visible through attrition or morale surveys — a toil percentage climbing past 50% for several consecutive weeks is a leading indicator that the team's operational load has outgrown its automation, and is a far earlier signal than waiting for someone to actually quit or burn out visibly.

The gotcha: teams under pressure to 'fix on-call burnout' often reach first for the easiest lever — hire more people, widen the rotation — because it's organizationally simple to approve and feels proactive, but if the underlying page volume and alert quality don't improve, the same number of low-value pages simply gets divided across a larger group, which measurably reduces each individual's page count without addressing why those pages exist in the first place, and the team is right back where it started the moment headcount growth stalls or the service scales up traffic again.

The measurement point defines the denominator. Server-side (app metrics): cheapest, richest labels — but requests that never reached the app (crashed pods, connection-refused, LB misroutes, DNS failures) simply don't exist in the data, so your 'availability' excludes exactly the worst outages; a service that's down reports no errors. LB/gateway-side: sees every request that reached your edge including 5xx from dead backends and timeouts — the sweet spot of coverage vs control for most services; this is where the SLO's error and latency SLIs should usually live. Client-side (RUM) and synthetics: capture DNS, CDN, TLS, last-mile — the real user truth — but RUM mixes in user networks/devices you can't fix, so it's better for tracking than for paging; synthetics give controlled outside-in coverage for the paths RUM can't (before launch, low-traffic endpoints) and catch 'green dashboards, site down' failures. Practical answer: page on LB-measured multi-window burn rate, run synthetics as the independent check, watch RUM for trend truth, and keep server-side metrics for debugging. And measure latency as a distribution at the same point — a p99 SLI computed from per-pod averages is fiction.

Definition discipline matters because everything unpleasant gets called toil: the test is manual + repetitive + automatable + tactical + no enduring value + O(service growth). Overhead (meetings, planning) isn't toil; novel incident response isn't toil (it's unplanned but not repetitive); the fifth identical certificate rotation absolutely is. Measurement: tag every ticket/page/interrupt with a toil category at close (same one-click discipline as alert dispositions); run quarterly time-sampling surveys as the cross-check (ticket data undercounts untracked interrupts); report toil-percentage per team with trend lines. The 50% ceiling (Google's canon) is a circuit breaker: above it, the team is servicing scale instead of engineering it away, and the number becomes an argument for headcount or roadmap change. Prioritization is expected-value arithmetic: annual occurrences x minutes x (1 + growth rate), against automation cost — plus risk-weighting for toil that occurs during incidents (3am toil compounds into outages via human error). One more honest input: automation that removes toil for the automating team but creates review burden elsewhere hasn't removed toil, it's exported it — count the system, not the team.

Three structural problems: (1) statistical — incident durations are log-normal-ish with heavy tails; the mean tracks your worst incident, not your typical one, so 'MTTR improved 30%' usually means 'we had fewer monsters this quarter', which is luck, not capability; (2) aggregation — time-to-detect (monitoring quality), time-to-engage (paging/escalation), time-to-mitigate (runbooks, rollback tooling), and time-to-full-repair are owned by different systems; one blended number can't tell you which to invest in; (3) incentive — a single headline number rewards closing tickets over restoring service. The better dashboard: per-severity percentiles (median and p90) of each phase; 'customer-first detection' count (should trend to zero); mitigation-vs-repair separated (mitigate fast, repair calmly); recurrence rate (same root-cause class within 90 days) as the counterweight — a team that mitigates in 5 minutes but has the same incident weekly is failing; and postmortem action-item completion within SLA as the learning-loop health check. Present distributions, not means, in ops reviews. The mature framing: incident speed metrics measure your response machinery; recurrence and action-completion measure whether the organization learns.

They nest: you measure an SLI (success rate, latency), set an SLO above what you promise to keep margin, and the SLA is the commercial agreement with consequences. Example: SLI current availability 99.95%; SLO target 99.9%; SLA promises 99.5% or credits apply. The gap between the SLO and 100% is the error budget that governs release risk.

The error budget turns reliability into a shared, objective decision tool that balances feature velocity against stability. Teams spend the budget on releases and risk; when it runs out, a change freeze (feature deployments stop, only stability work) protects the SLO. This aligns dev and ops around one number instead of arguing about "how much testing is enough".

Metrics quantify system state over time and drive alerts; logs give event-level detail for a component; distributed traces follow one request end to end to localize latency or errors across services. Correlating a trace ID into logs links them. Observability (explore unknown-unknowns) is broader than monitoring (watch known conditions).

Monitoring generates an alert; responders detect and acknowledge, then mitigate to restore service (rollback, failover, scale) before chasing root cause. After recovery, do RCA (e.g., 5 Whys) and a blameless postmortem documenting timeline, contributing factors, and owned action items. The goal is to reduce impact fast and prevent recurrence, not to assign blame.

Each answer becomes the next question, drilling past symptoms to the systemic cause so the fix prevents recurrence rather than patching the surface. It works best for linear cause chains; complex incidents may need broader techniques (fishbone, contributing-factors analysis). Pair it with a blameless postmortem so people answer honestly.

Change freezes cut the biggest source of incidents — change — during high-risk periods (holidays, launches, on-call gaps). A hard freeze = no changes, no deployments, maximum stability. A cold freeze = critical fixes and security patches only, with everything else held. Define the window, the exception process, and who approves emergency changes.

Google defines toil as work that is manual, repetitive, automatable, tactical, and O(n) with growth. Left unchecked it consumes the team and prevents reliability engineering. The response: measure toil, keep it under a budget (often ~50%), and automate the recurring tasks (scripts, operators, self-healing) so engineers work on durable improvements.