What is automatic mixed precision (AMP) and what are the risks of enabling it blindly?
Quick Answer
AMP runs eligible ops in float16/bfloat16 while keeping numerically sensitive ops in float32, roughly halving memory and often near-doubling throughput on tensor cores. Risks: float16 gradient underflow (needs GradScaler), loss spikes on some models, and small numerical drift that can change model behavior.
Detailed Answer
torch.autocast wraps forward/loss computation and dispatches each op to a precision from an allow-list: matmuls and convolutions in half precision, reductions like softmax/layernorm in float32. With float16 you also need torch.cuda.amp.GradScaler, which scales the loss up before backward so tiny gradients don't underflow to zero, then unscales before the optimizer step and skips steps with inf/NaN. bfloat16 (Ampere+) has float32's exponent range, so it usually needs no scaler and is the default choice today. The risks: numerical drift means training curves and even inference outputs are not bit-identical to float32 — problematic for regression-tested pipelines; some architectures (large activations, certain attention patterns) hit NaN losses and need selective float32 casting; and validation done in a different precision than serving can hide accuracy gaps. Enable AMP with a controlled A/B on final metrics, not just speed.
Code Example
scaler = torch.cuda.amp.GradScaler()
for x, y in loader:
with torch.autocast('cuda', dtype=torch.bfloat16):
loss = criterion(model(x), y)
scaler.scale(loss).backward()
scaler.step(optimizer)
scaler.update()
optimizer.zero_grad(set_to_none=True)Interview Tip
Say why bfloat16 mostly replaced float16 (same exponent range as fp32, so no GradScaler gymnastics) — it's a one-liner that dates your knowledge as current.