How do DataLoader num_workers and pin_memory actually affect training throughput, and what goes wrong when you misconfigure them?
Quick Answer
num_workers>0 forks worker processes that prefetch and transform batches in parallel so the GPU never starves; pin_memory=True stages batches in page-locked RAM enabling faster async host-to-GPU copies. Too many workers cause RAM blow-up, shared-memory errors, and fork overhead.
Detailed Answer
With num_workers=0 the main process loads data synchronously — the GPU idles during every batch load, and you will see low GPU utilization with high step time. Workers prefetch batches in parallel (each holds prefetch_factor batches in RAM), which usually fixes GPU starvation. Misconfiguration symptoms: too many workers multiply dataset memory (each worker gets a copy-on-write fork of the dataset object — deadly if the dataset holds a big in-RAM array), exhaust /dev/shm in containers (the classic 'DataLoader worker (pid X) is killed by signal: Bus error' in Kubernetes pods with the default 64MB shm), and add fork startup cost every epoch unless persistent_workers=True. pin_memory=True allocates the staging buffers in page-locked memory so .to('cuda', non_blocking=True) overlaps copy with compute; it costs host RAM and does nothing for CPU training. A sane starting point: num_workers = min(4-8, cores per GPU), pin_memory=True on GPU, persistent_workers=True, then profile — the right number is workload-specific.
Code Example
loader = DataLoader(ds, batch_size=128, num_workers=8,
pin_memory=True, persistent_workers=True,
prefetch_factor=4)
for x, y in loader:
x = x.to('cuda', non_blocking=True) # overlaps with compute
# K8s: shm too small for workers
# volumes: emptyDir: {medium: Memory} mounted at /dev/shmInterview Tip
Mentioning the Kubernetes /dev/shm bus error is a strong production signal — it's the single most common PyTorch-in-containers gotcha and most candidates have never heard of it.