PyTorch

model.eval() is about layer semantics: dropout stops dropping activations and batch norm uses its running statistics instead of batch statistics. It does nothing to autograd — PyTorch will still record every operation for backprop, allocating memory for intermediate activations. torch.no_grad() (or the stricter torch.inference_mode()) is about memory and speed: it tells autograd not to build the computation graph at all, cutting inference memory roughly in half and speeding up the forward pass. Forgetting eval() gives you wrong (noisy or shifted) predictions; forgetting no_grad() gives you correct predictions that slowly eat your GPU memory under load. In serving code, wrap the model call in torch.inference_mode() and call model.eval() once at load time.

OOM-after-hours is almost never the model itself — it loaded fine. The usual suspects, in order: (1) missing torch.no_grad(), so every request builds an autograd graph that is retained as long as the output tensor lives; (2) Python references keeping tensors alive — appending raw CUDA tensors to a metrics list or cache instead of calling .item() or .cpu(); (3) fragmentation — allocated memory is fine but there is no contiguous block big enough, common with highly variable input shapes; (4) unbounded inputs — one client sends a 10x-longer sequence and peak memory spikes. Diagnose with torch.cuda.memory_allocated() vs memory_reserved() over time (steady climb = leak; flat allocated but OOM = fragmentation), torch.cuda.memory_summary(), and the memory snapshot profiler. Fixes map one-to-one: inference_mode, .item()/.detach().cpu() on anything you keep, expandable_segments:True or bucketed padding for fragmentation, and hard caps on batch/sequence size at the API layer.

torch.compile captures the Python-level model code into an FX graph (TorchDynamo), then compiles fused GPU kernels (TorchInductor), eliminating Python overhead and kernel launch latency. As of PyTorch 2.6+ it is production-stable and widely used for LLM inference, often combined with CUDA graphs. But it has real costs: a 30-60 second warm-up compile for a 7B model, cache invalidation on shape changes, and recompilation whenever guard conditions fail. Do not use it when (1) request shapes vary wildly and you cannot bucket-pad — recompilation overhead can exceed the speedup; (2) the model uses custom ops Dynamo cannot trace, so graph breaks fall back to eager and you keep the complexity without the gains; (3) you already serve through TensorRT-LLM or vLLM, which manage their own compilation; (4) cold-start latency matters more than steady-state throughput (e.g., scale-to-zero serverless). Also note a documented gotcha: torch.compile can create a CUDA context even for CPU-only code paths, which surprises multi-process CPU deployments.

DataParallel (DP) replicates the model onto each GPU inside a single process, splits each batch, and gathers outputs back on the primary GPU every forward pass. That design has three structural problems: the Python GIL serializes coordination, model replication happens every iteration, and GPU-0 becomes a memory/computation hotspot. DistributedDataParallel (DDP) launches one process per GPU (torchrun), each with its own model replica; gradients are synchronized with NCCL all-reduce, overlapped with the backward pass via gradient bucketing. DDP is faster even on a single machine, works across nodes, and composes with mixed precision and torch.compile. In practice DP survives only in quick notebook experiments; anything that matters uses DDP or, for models too big for one GPU, FSDP (fully sharded data parallel). Interview-wise, also know that DDP requires DistributedSampler so each rank sees a distinct data shard.

With num_workers=0 the main process loads data synchronously — the GPU idles during every batch load, and you will see low GPU utilization with high step time. Workers prefetch batches in parallel (each holds prefetch_factor batches in RAM), which usually fixes GPU starvation. Misconfiguration symptoms: too many workers multiply dataset memory (each worker gets a copy-on-write fork of the dataset object — deadly if the dataset holds a big in-RAM array), exhaust /dev/shm in containers (the classic 'DataLoader worker (pid X) is killed by signal: Bus error' in Kubernetes pods with the default 64MB shm), and add fork startup cost every epoch unless persistent_workers=True. pin_memory=True allocates the staging buffers in page-locked memory so .to('cuda', non_blocking=True) overlaps copy with compute; it costs host RAM and does nothing for CPU training. A sane starting point: num_workers = min(4-8, cores per GPU), pin_memory=True on GPU, persistent_workers=True, then profile — the right number is workload-specific.

torch.autocast wraps forward/loss computation and dispatches each op to a precision from an allow-list: matmuls and convolutions in half precision, reductions like softmax/layernorm in float32. With float16 you also need torch.cuda.amp.GradScaler, which scales the loss up before backward so tiny gradients don't underflow to zero, then unscales before the optimizer step and skips steps with inf/NaN. bfloat16 (Ampere+) has float32's exponent range, so it usually needs no scaler and is the default choice today. The risks: numerical drift means training curves and even inference outputs are not bit-identical to float32 — problematic for regression-tested pipelines; some architectures (large activations, certain attention patterns) hit NaN losses and need selective float32 casting; and validation done in a different precision than serving can hide accuracy gaps. Enable AMP with a controlled A/B on final metrics, not just speed.

For CPU inference the wins come from (1) getting out of eager Python execution, (2) quantization, and (3) thread discipline. Options: ONNX Runtime — export once, get graph fusions and a mature int8 quantization toolchain; usually the best latency/effort ratio and language-agnostic serving. torch.export + AOTInductor (the modern TorchScript replacement) — ahead-of-time compiles to a shared library loadable from C++ without the Python interpreter. TorchScript still works but is de-emphasized. Dynamic int8 quantization typically gives 2-4x on linear-heavy models with small accuracy cost — validate on a golden set. Thread control matters more than people expect: the default 'use all cores' setting causes thrashing when multiple workers share a box — set torch.set_num_threads / OMP_NUM_THREADS so workers x threads = physical cores. Finally decide batching policy: micro-batching helps throughput but hurts p99; for strict p99, batch size 1 with more replicas is often correct.