What is the difference between DataParallel and DistributedDataParallel, and why is DataParallel effectively deprecated?
Quick Answer
DataParallel is single-process multi-GPU: one Python process scatters batches and gathers results each step, bottlenecked by the GIL and GPU-0. DistributedDataParallel runs one process per GPU with all-reduce gradient sync — faster, scales across nodes, and is the only option the team recommends.
Detailed Answer
DataParallel (DP) replicates the model onto each GPU inside a single process, splits each batch, and gathers outputs back on the primary GPU every forward pass. That design has three structural problems: the Python GIL serializes coordination, model replication happens every iteration, and GPU-0 becomes a memory/computation hotspot. DistributedDataParallel (DDP) launches one process per GPU (torchrun), each with its own model replica; gradients are synchronized with NCCL all-reduce, overlapped with the backward pass via gradient bucketing. DDP is faster even on a single machine, works across nodes, and composes with mixed precision and torch.compile. In practice DP survives only in quick notebook experiments; anything that matters uses DDP or, for models too big for one GPU, FSDP (fully sharded data parallel). Interview-wise, also know that DDP requires DistributedSampler so each rank sees a distinct data shard.
Code Example
# torchrun --nproc_per_node=4 train.py
dist.init_process_group('nccl')
rank = dist.get_rank()
model = DDP(model.to(rank), device_ids=[rank])
sampler = DistributedSampler(dataset)
loader = DataLoader(dataset, sampler=sampler, batch_size=64)Interview Tip
Name the three DP bottlenecks (GIL, per-step replication, GPU-0 hotspot) and mention FSDP as the answer for models that don't fit on one GPU — that shows you know the current landscape, not 2019's.