How do you set up on-call rotations and incident response runbooks for a Kubernetes platform team?

Think of on-call like a hospital emergency room. There is always a triage nurse (first responder), a resident doctor (secondary on-call), and an attending physician (escalation). Each has a clear role, and they follow standardized protocols for common emergencies. You would never want a nurse encountering a cardiac arrest to have to figure out the procedure from scratch — they follow a runbook. Kubernetes on-call works the same way: clear rotations, defined escalation paths, and actionable runbooks that turn any engineer into an effective incident responder.

On-call rotations for a banking platform team should be structured in tiers. Tier 1 is the primary on-call engineer who receives all alerts and performs initial triage. Tier 2 is a secondary on-call (typically a more senior engineer) who gets escalated to if the primary cannot resolve the issue within 30 minutes. Tier 3 is the team lead or architect for critical incidents that require architectural decisions or cross-team coordination. Rotations typically last one week, with handoff meetings every Monday where the outgoing on-call briefs the incoming on-call on active issues, recent deployments, and any known risks. In banking, you must also have coverage for regulatory reporting windows — the ACH processing team may need dedicated on-call coverage during settlement windows that differ from standard business hours.

PagerDuty (or Opsgenie) configuration should include escalation policies, scheduling overrides for holidays and vacations, and integration with your alerting stack. Prometheus alerts fire to Alertmanager, which routes to PagerDuty based on severity labels. SEV1 alerts (payments-api completely down) page immediately with phone calls. SEV2 alerts (degraded performance on settlements-processor) send push notifications with a 15-minute acknowledgment window before escalating. SEV3 alerts (non-critical capacity warnings) create Slack messages in the platform-ops channel for business-hours review. The key is alert fatigue prevention — if your on-call engineer receives more than 2-3 pages per shift on average, you have a signal-to-noise problem that must be addressed before it leads to burnout and missed critical alerts.

Runbooks are the single most impactful investment for on-call effectiveness. Every Prometheus alert should have a runbook_url annotation that links to a version-controlled runbook in your Git repository. A good runbook follows a template: alert description (what this alert means in business terms), impact assessment (which customers or services are affected), diagnostic steps (specific kubectl and Prometheus commands to run), remediation steps (exact actions to fix the issue, ordered from least to most disruptive), escalation criteria (when to wake up the secondary on-call or declare a major incident), and communication template (what to post in Slack and who to notify). In a banking context, runbooks must also include regulatory notification requirements — certain types of outages must be reported to regulators within specific timeframes.

In production, runbooks are living documents that evolve after every incident. The post-mortem process should always review whether the relevant runbook existed, was accurate, and was followed. If the on-call engineer had to improvise, the runbook gets updated as a post-mortem action item. Some teams implement runbook automation using tools like Rundeck or PagerDuty Automation Actions, where common remediation steps (like restarting a crashed pod, scaling up replicas, or clearing a queue backlog) can be triggered directly from the PagerDuty alert with a single click. This reduces mean time to recovery and makes runbooks actionable even for junior engineers who may not be comfortable running kubectl commands under pressure.

The biggest gotcha is treating on-call as an individual burden rather than a team responsibility. On-call compensation (either monetary or time-off-in-lieu) is essential for sustainability. Engineers should never be on-call for more than one week per month, and follow-the-sun rotations across time zones are preferable to 24/7 coverage from a single location. Another common mistake is runbooks that are too long or too vague — a runbook that says 'investigate the issue' is useless at 3 AM. Keep runbooks to one page, with copy-pasteable commands and clear decision trees. Finally, run regular 'game day' exercises where you simulate incidents and have engineers follow runbooks to validate they work and identify gaps.