What is the difference between torch.no_grad() and model.eval(), and why do you need both during inference?
Quick Answer
model.eval() switches layers like dropout and batch norm into inference behavior; torch.no_grad() disables autograd graph construction so no gradient buffers are allocated. They solve different problems — production inference should use both.
Detailed Answer
model.eval() is about layer semantics: dropout stops dropping activations and batch norm uses its running statistics instead of batch statistics. It does nothing to autograd — PyTorch will still record every operation for backprop, allocating memory for intermediate activations. torch.no_grad() (or the stricter torch.inference_mode()) is about memory and speed: it tells autograd not to build the computation graph at all, cutting inference memory roughly in half and speeding up the forward pass. Forgetting eval() gives you wrong (noisy or shifted) predictions; forgetting no_grad() gives you correct predictions that slowly eat your GPU memory under load. In serving code, wrap the model call in torch.inference_mode() and call model.eval() once at load time.
Code Example
model = MyModel()
model.load_state_dict(torch.load('weights.pt', map_location='cuda'))
model.eval() # once, at load time
@torch.inference_mode() # every request
def predict(batch):
return model(batch.to('cuda'))Interview Tip
Most candidates know one of the two. Saying "eval() is layer semantics, no_grad() is autograd memory — production needs both" in the first sentence signals you have actually served models, not just trained them.