How do you design an on-call rotation that avoids burnout, and why does simply adding more people to the rotation not fix a broken on-call system by itself?
Quick Answer
A sustainable rotation needs a minimum team size (commonly 8+ for full 24x7 coverage), a primary-plus-secondary escalation structure, alerts tiered so only genuine customer-impacting issues page a human, and an explicit toil budget — tracking how much on-call time goes to repetitive manual work versus real engineering, with automation prioritized once toil crosses roughly 50%. Adding more people spreads the same number of low-value pages across more humans, which reduces individual burnout risk but does nothing to fix the underlying alert quality problem that's actually causing burnout in the first place.
Detailed Answer
Imagine a fire department that gets called out constantly for burst water pipes, not actual fires — adding more firefighters to the roster does reduce how often any single firefighter gets called, but it doesn't address the real problem, which is that most of the calls shouldn't require a firefighter at all. On-call burnout works exactly the same way: the actual cause is almost never 'too few people in the rotation,' it's alerts that page a human for things that don't need urgent human judgment — noisy, low-value pages that erode trust in the alerting system and train people to view every page with dread rather than as a meaningful signal.
This is why the on-call design problem should be tackled as an alert-quality problem first and a staffing-math problem second. Google's SRE practice frames this explicitly through the concept of toil — repetitive, manual, automatable operational work that scales linearly with service growth rather than getting cheaper over time — and recommends capping toil at roughly 50% of an SRE's time specifically because toil that exceeds that threshold means the team is spending more time reacting to the same categories of problems than it is spending building the automation that would prevent those problems from recurring, a trap that gets structurally worse, not better, the longer it's tolerated.
Step by step, a well-designed rotation layers several independent safeguards: first, alert tiering — only genuinely customer-impacting symptoms (elevated error rate, breached latency SLO, full service unavailability) should page a human immediately; internal, non-urgent signals (a single retryable failed job, a metric drifting but still within a safe range) should generate a ticket or a dashboard entry, not a 2 AM page. Second, a primary-plus-secondary structure means a single missed or delayed acknowledgment doesn't leave an incident unattended — the secondary is a deliberate redundancy, not just a backup name on a list nobody expects to actually page. Third, hard caps on pages per shift (a common target is fewer than two genuine pages per shift) act as a forcing function: if a service consistently exceeds that cap, that's treated as a signal the alerting or the service itself needs engineering attention, not just something to individually tolerate. Fourth, blameless postmortems — analyzing what happened and why without assigning individual fault — exist because on-call engineers who fear blame for an incident start hiding problems or delaying escalation, which directly undermines the psychological safety needed for people to sustainably volunteer for high-stakes rotations at all.
At organizational scale, teams that track toil explicitly (logging roughly how much on-call time each week goes to manual, repeatable interventions versus genuine incident response or engineering work) get an early, objective warning sign long before individual burnout becomes visible through attrition or morale surveys — a toil percentage climbing past 50% for several consecutive weeks is a leading indicator that the team's operational load has outgrown its automation, and is a far earlier signal than waiting for someone to actually quit or burn out visibly.
The gotcha: teams under pressure to 'fix on-call burnout' often reach first for the easiest lever — hire more people, widen the rotation — because it's organizationally simple to approve and feels proactive, but if the underlying page volume and alert quality don't improve, the same number of low-value pages simply gets divided across a larger group, which measurably reduces each individual's page count without addressing why those pages exist in the first place, and the team is right back where it started the moment headcount growth stalls or the service scales up traffic again.