How do you reduce toil in Kubernetes operations — what repetitive tasks do you automate first?

Imagine you are a bank teller who spends four hours every day manually stamping forms that could be processed by a machine. That stamping is toil — it is manual, repetitive, automatable, and it scales linearly with the number of forms. In SRE, toil is operational work that is manual, repetitive, automatable, tactical, lacks enduring value, and grows with the size of the service. Google's SRE book recommends keeping toil below 50% of an SRE team's time, with the other 50% spent on engineering projects that permanently reduce future toil.

The first step is measuring toil. Have each team member track their operational tasks for two weeks using time categories: incident response, manual deployments, certificate management, access requests, capacity adjustments, and ad-hoc troubleshooting. In a banking Kubernetes environment, common toil sources include manually creating namespaces and RBAC roles for new applications, rotating TLS certificates, patching worker nodes, responding to disk pressure alerts, scaling deployments for end-of-month batch processing, and generating compliance reports. Once you have data, rank tasks by frequency multiplied by time-per-occurrence to find the highest-impact automation targets.

Certificate management is usually the first high-value target. In a bank running payments-api and settlements-processor, you might have dozens of internal TLS certificates that expire every 90 days. Installing cert-manager with a ClusterIssuer connected to your internal CA (or Let's Encrypt for external services) eliminates this entirely — certificates are automatically provisioned, renewed 30 days before expiry, and stored as Kubernetes secrets. Node patching is another major toil source. Kured (Kubernetes Reboot Daemon) detects when nodes need a reboot after OS updates and performs rolling reboots during maintenance windows, draining pods gracefully before rebooting. For EKS, managed node group updates can be automated with a Terraform pipeline that updates the AMI version and performs rolling replacements.

Namespace and RBAC provisioning is a significant toil source in banking environments where every new microservice needs a namespace with network policies, resource quotas, service accounts, and role bindings that comply with PCI-DSS requirements. Building a Kubernetes operator (or using Crossplane Compositions) that takes a simple custom resource as input and creates the entire compliant namespace setup eliminates hours of manual YAML crafting and review. Developers submit a pull request with a TeamNamespace custom resource, and the operator provisions everything — including Vault integration for secrets, OPA policies for pod security, and Prometheus ServiceMonitors for monitoring.

In production, the most impactful toil reduction often comes from self-service platforms. Instead of developers filing Jira tickets for the SRE team to create a database, provision a Redis cache, or update a DNS record, build Backstage templates or Crossplane claims that let developers provision these resources through a catalog with guardrails. The SRE team defines the templates with built-in compliance (encryption at rest, backup schedules, audit logging), and developers consume them without needing to understand the underlying infrastructure. This shifts the SRE team from being a ticket-processing bottleneck to being platform engineers who build the guardrails once.

The biggest gotcha is automating something that should be eliminated instead. Before automating a process, ask: 'Should this process exist at all?' If your team manually approves every deployment to staging, maybe the answer is not to automate the approval but to remove it entirely and rely on automated tests and canary analysis. Another trap is building custom automation that duplicates existing tools — before writing a Python script to rotate secrets, check if External Secrets Operator already does this. Finally, measure the impact of your automation. Track toil hours per week as a team metric, and ensure your automation investments are actually reducing them rather than just creating new maintenance burden.