Netflix platform interview: How would you build a Python Kubernetes watcher that detects…

Netflix platform interview: How would you build a Python Kubernetes watcher that detects CrashLoopBackOff pods and creates a structured incident event without missing updates or hammering the API server?

⚡

Quick Answer

Use the official Kubernetes Python client, load in-cluster or kubeconfig credentials, list with resourceVersion, then watch pod events with timeouts and reconnect logic. Filter pod container statuses for waiting.reason equals CrashLoopBackOff, deduplicate by namespace/name/container/restart count, and emit structured events instead of polling every few seconds.

Detailed Answer

A Kubernetes watcher is like a security desk watching badge events at a large office. Asking for the full list every five seconds is wasteful and still misses context. Subscribing to a stream and remembering the last seen position is cheaper and more accurate. That is the mental model interviewers want when they ask about Kubernetes automation in Python.

The official Kubernetes Python client exposes CoreV1Api and watch.Watch. A production watcher usually starts with a list_namespaced_pod or list_pod_for_all_namespaces call to get the current resourceVersion, then uses watch.stream against the same list function. The resourceVersion is important because Kubernetes objects are versioned; it lets the watcher resume from a known point instead of starting from scratch every time.

CrashLoopBackOff is not a pod phase. It appears in container status under state.waiting.reason, often alongside restartCount and lastState. Good code inspects initContainerStatuses and containerStatuses, records the exact container, and includes namespace, pod name, node, image, restart count, and last termination reason. It deduplicates so one broken pod does not create 500 Slack messages.

At scale, the watcher must respect API server limits. Use watch timeouts so the HTTP stream refreshes periodically. Handle 410 Gone by relisting because the resourceVersion is too old. Avoid one watcher per namespace unless needed; a single all-namespaces watch is usually simpler. Emit metrics for reconnects, event lag, and handler errors so the automation itself is observable.

The non-obvious gotcha is RBAC and blast radius. A watcher running in-cluster should use a ServiceAccount with list/watch on pods only, not cluster-admin. If it creates incident objects, write those to a narrow namespace or external queue, not back into every application namespace.

from __future__ import annotations # Use modern Python annotations. from kubernetes import client, config, watch # Import the official Kubernetes Python client. from kubernetes.client.exceptions import ApiException # Handle Kubernetes API errors. def load_config() -> None: # Load credentials for local or in-cluster execution. try: # Prefer in-cluster configuration for deployed watchers. config.load_incluster_config() # Reads service account token and CA from the pod. except config.ConfigException: # Fall back when running locally. config.load_kube_config() # Reads the user's kubeconfig file. def crashloop_events() -> None: # Stream CrashLoopBackOff signals. load_config() # Configure Kubernetes client authentication. api = client.CoreV1Api() # Create Core API client for pods. resource_version = "" # Empty means start from the latest list result. seen: set[tuple[str, str, str, int]] = set() # Deduplicate repeated watch events. while True: # Reconnect forever like a controller loop. try: # Keep API errors from killing the process. stream = watch.Watch().stream( # Start a Kubernetes watch stream. api.list_pod_for_all_namespaces, # Watch pods across namespaces. resource_version=resource_version, # Resume from the last version when possible. timeout_seconds=300, # Refresh the HTTP watch periodically. ) # The stream yields ADDED, MODIFIED, and DELETED events. for event in stream: # Process each watched event. pod = event["object"] # Extract the V1Pod object. resource_version = pod.metadata.resource_version # Save progress for reconnect. statuses = pod.status.container_statuses or [] # Regular containers may be absent early. for status in statuses: # Inspect each container status. state = status.state.waiting if status.state else None # CrashLoopBackOff appears in waiting state. if not state or state.reason != "CrashLoopBackOff": # Ignore healthy or unrelated states. continue # Move to the next container. key = (pod.metadata.namespace, pod.metadata.name, status.name, status.restart_count) # Build idempotency key. if key in seen: # Avoid duplicate incident notifications. continue # Skip already emitted restart count. seen.add(key) # Mark this signal as emitted. print({ # Replace print with Kafka, webhook, or incident API in production. "namespace": pod.metadata.namespace, # Include namespace for routing. "pod": pod.metadata.name, # Include pod name. "container": status.name, # Include failing container. "node": pod.spec.node_name, # Include node for correlated incidents. "restarts": status.restart_count, # Include restart count for severity. "image": status.image, # Include deployed image. }) # Structured event payload. except ApiException as exc: # Kubernetes returned an API error. if exc.status == 410: # resourceVersion too old. resource_version = "" # Relist from current state. continue # Restart the watch loop. raise # Unexpected API errors should be surfaced.

Netflix platform interview: How would you build a Python Kubernetes watcher that detects CrashLoopBackOff pods and creates a structured incident event without missing updates or hammering the API server?

More Python DevOps Automation questions