How do you measure toil, and how do you decide how much engineering time to spend automating it away?
Quick Answer
Toil is manual, repetitive, automatable, reactive work that scales with service growth — measure it by tagging tickets/interrupts and periodic time-tracking sampling, targeting the classic <50% of SRE time bound. Prioritize automation by frequency x time-per-occurrence x growth trajectory, and fund it explicitly — toil above the bound is a staffing/roadmap decision, not a personal failing.
Detailed Answer
Definition discipline matters because everything unpleasant gets called toil: the test is manual + repetitive + automatable + tactical + no enduring value + O(service growth). Overhead (meetings, planning) isn't toil; novel incident response isn't toil (it's unplanned but not repetitive); the fifth identical certificate rotation absolutely is. Measurement: tag every ticket/page/interrupt with a toil category at close (same one-click discipline as alert dispositions); run quarterly time-sampling surveys as the cross-check (ticket data undercounts untracked interrupts); report toil-percentage per team with trend lines. The 50% ceiling (Google's canon) is a circuit breaker: above it, the team is servicing scale instead of engineering it away, and the number becomes an argument for headcount or roadmap change. Prioritization is expected-value arithmetic: annual occurrences x minutes x (1 + growth rate), against automation cost — plus risk-weighting for toil that occurs during incidents (3am toil compounds into outages via human error). One more honest input: automation that removes toil for the automating team but creates review burden elsewhere hasn't removed toil, it's exported it — count the system, not the team.
Code Example
toil_score = freq_per_year * minutes_each * (1 + yoy_growth) / 60 # hours/yr backlog (sorted): cert-rotations: 52 * 45 * 1.0 = 39h/yr -> automate (2d effort) db-failover-drill: 4 * 240 * 1.0 = 16h/yr -> keep manual (practice value) stale-node-cleanup: 180 * 12 * 1.4 = 50h/yr -> automate first
Interview Tip
Cite the definition tests (not everything unpleasant is toil) and the 50% bound as a circuit breaker that triggers staffing conversations — that's the canonical-but-practiced answer. The 'exported toil' caveat earns senior points.