What are taints, tolerations, and node affinity -- how do they control Pod scheduling?

Imagine an apartment building where some floors have special restrictions. Taints are like signs on certain floors saying 'No Pets Allowed' or 'Residents Only.' Tolerations are like having a special pet permit that lets you live on a no-pets floor despite the restriction. Node affinity is like stating a preference on your housing application: 'I strongly prefer a floor with a balcony' (required) or 'I would like a south-facing unit if possible' (preferred). The building manager (scheduler) uses all three to decide which unit (node) each new resident (Pod) gets.

In Kubernetes, taints are key-value pairs with an effect applied to nodes using kubectl taint nodes. There are three effects: NoSchedule (new Pods without a matching toleration will not be scheduled), PreferNoSchedule (the scheduler tries to avoid the node but will use it if necessary), and NoExecute (existing Pods without a matching toleration are evicted, and new ones are not scheduled). Tolerations are defined in the Pod spec and must match the taint's key, value, and effect. The operator can be Equal (exact match on key and value) or Exists (matches any value for the key). Kubernetes automatically adds taints for node conditions like NotReady, Unreachable, MemoryPressure, DiskPressure, and PIDPressure.

Node affinity, defined in spec.affinity.nodeAffinity, comes in two forms. RequiredDuringSchedulingIgnoredDuringExecution is a hard constraint: the Pod will only be scheduled on nodes matching the selector, and it stays Pending if no matching node exists. PreferredDuringSchedulingIgnoredDuringExecution is a soft constraint: the scheduler prefers matching nodes but will place the Pod elsewhere if needed, with a weight (1-100) influencing how much the preference affects the scoring phase. The 'IgnoredDuringExecution' suffix means that if a node's labels change after the Pod is running, the Pod is not evicted. There is a planned requiredDuringSchedulingRequiredDuringExecution that would evict Pods when nodes no longer match, but it is not yet implemented.

In production, these mechanisms are commonly layered together. GPU nodes are tainted with nvidia.com/gpu=present:NoSchedule so only ML workloads with the matching toleration land there, preventing general Pods from wasting expensive GPU resources. Node affinity is used to target specific instance types (node.kubernetes.io/instance-type: p3.8xlarge) or availability zones (topology.kubernetes.io/zone: us-east-1a) for data locality. The combination of taints and node affinity creates both push and pull forces: taints push non-matching Pods away, and affinity pulls matching Pods toward specific nodes. Pod anti-affinity (spec.affinity.podAntiAffinity) complements these by spreading Pods of the same type across different nodes or zones.

A non-obvious gotcha is that tolerations allow but do not require scheduling on a tainted node. A Pod that tolerates a GPU taint might still land on a non-GPU node if the scheduler scores it higher. You need both a toleration (to permit) and node affinity (to attract) to ensure Pods land on specific nodes. Another trap: the NoExecute taint effect has an optional tolerationSeconds field that evicts the Pod after a grace period. Kubernetes adds NoExecute taints for NotReady nodes with a default toleration of 300 seconds on all Pods, meaning Pods tolerate node failures for 5 minutes before being rescheduled. If your application is latency-sensitive, you may want to reduce this with a custom toleration. Also, taints are not visible in kubectl get nodes output by default -- use kubectl describe node or kubectl get nodes -o json to audit taints, which is why misconfigured taints often go undetected.