A node running as a metrics aggregator has a Pod that becomes unresponsive and not ready. How does Kubernetes handle the storage attachment and how do you recover safely without data corruption?

Think of this like a locked filing cabinet. Only one person can have the key at a time. If the person holding the key (the stuck Pod) becomes unresponsive, you can't just give a copy of the key to someone else — that would mean two people writing to the same files simultaneously, causing corruption. You have to get the original key back first (confirm the original Pod is fully stopped), then hand it to the replacement.

When a Pod using a ReadWriteOnce PersistentVolume becomes not ready but the node is still running, here's the sequence: the kubelet marks the Pod as not ready, the readiness probe fails, and the Pod is removed from Service endpoints. However, the volume remains attached to the node and mounted in the Pod's filesystem. Kubernetes will not detach a volume from a node while any Pod on that node still references it — even if that Pod is in a bad state.

If the Pod is stuck in Terminating (the node acknowledges the delete but the process won't stop), you can try kubectl delete pod --grace-period=0 --force. But this only removes the Pod from the API server — it does NOT guarantee the process is stopped on the node. The kubelet still needs to kill the container. If the node itself is unreachable (network partition, kernel hang), the Pod stays in Terminating indefinitely because the API server can't confirm the container is gone.

This is where it gets dangerous: if you force-delete the Pod and the scheduler creates a replacement on a different node, Kubernetes tries to attach the volume to the new node. But the volume is still attached to the old node. Cloud providers (AWS, GCP) will deny the attach because the volume is in-use. The new Pod sits in ContainerCreating with a 'Multi-Attach error' until the old attachment is released. On AWS EBS, this force-detach can take 5-10 minutes, and if the old node was writing data at the moment of detach, you risk filesystem corruption.

The safe recovery process: (1) Confirm the old node's status — if the node is reachable, let the kubelet handle cleanup. (2) If the node is unreachable, verify the node is truly dead (not just network-partitioned). (3) Only after confirming the node won't write to the volume, delete the VolumeAttachment object or force-detach from the cloud console. (4) Wait for the volume to show as Available before letting the new Pod start. (5) After attachment, run filesystem checks if corruption is suspected.

For StatefulSets specifically, Kubernetes is extra cautious: it will NOT create a replacement Pod until the old Pod is confirmed deleted (including volume cleanup). This 'at-most-one' guarantee prevents dual-mount scenarios but means recovery takes longer.