How does Kubernetes handle graceful shutdown -- preStop hooks, SIGTERM, and terminationGracePeriod?

Think of closing a restaurant at the end of the night. You do not just cut the power -- first you stop seating new customers (remove from service), then you let the kitchen finish current orders (preStop hook), then you announce 'last call' to remaining diners (SIGTERM), give them 30 minutes to finish up (terminationGracePeriodSeconds), and if someone is still there after that, security escorts them out (SIGKILL). Each step is essential for a graceful wind-down that does not leave customers mid-meal.

In Kubernetes, the graceful shutdown sequence begins when the API server marks a Pod for deletion. This triggers two parallel processes: first, the Endpoints controller removes the Pod's IP from all Service Endpoints (and EndpointSlices), and second, the kubelet on the Pod's node begins the termination sequence. The parallel nature of these processes is the root cause of many production issues -- the kubelet may send SIGTERM before the Endpoints update has propagated to all kube-proxy instances and ingress controllers, meaning the Pod may still receive new requests after it has begun shutting down.

The kubelet's termination sequence works as follows: (1) Set the Pod status to Terminating. (2) Execute preStop hooks on all containers that define them. PreStop hooks can be either an exec command (running a script inside the container) or an HTTP GET request to an endpoint. PreStop hooks run concurrently across containers. (3) After all preStop hooks complete (or after a timeout), send SIGTERM (signal 15) to PID 1 of each container simultaneously. (4) Wait for all containers to exit, up to terminationGracePeriodSeconds (which counts from the start of step 2, not from SIGTERM). (5) If any container is still running when the grace period expires, send SIGKILL (signal 9) to forcefully terminate it. The total time for preStop plus SIGTERM handling must fit within terminationGracePeriodSeconds.

In production, the most common graceful shutdown failure is receiving requests after SIGTERM. This happens because the Endpoints update races with the kubelet's termination sequence. The industry-standard fix is to add a preStop hook with a sleep of 5-15 seconds, giving kube-proxy and ingress controllers time to update their routing rules before the application begins shutting down. During this sleep, the application still handles in-flight requests normally. After the sleep, when SIGTERM arrives, the application should stop accepting new connections, finish processing in-flight requests, close database connections and flush buffers, and then exit cleanly. Load balancers external to Kubernetes (like AWS ALB) have their own deregistration delays that must be coordinated with the terminationGracePeriodSeconds.

A non-obvious gotcha is that terminationGracePeriodSeconds includes preStop hook execution time. If your preStop hook takes 20 seconds and your grace period is 30 seconds, the application only has 10 seconds after SIGTERM to finish its work. Many teams set a generous grace period (120-300 seconds for long-running workloads like batch jobs) but forget that during rolling updates, old Pods must terminate within this period before new ones can take their place, slowing down deployments. Another trap: if your container's PID 1 process is a shell script that does not forward signals, SIGTERM is caught by the shell and never reaches the actual application. Use exec in your entrypoint script or use tini/dumb-init as PID 1 to ensure signal propagation. Finally, containers using the default /dev/termination-log file should write failure context there before exiting, as it surfaces in kubectl describe pod output and helps debug termination issues.